Add interactive chat with multi-provider LLM support (Ollama, llama.cpp, Ramalama, vLLM), powered by rig#87
Conversation
| AnyAgent::Ollama(builder.build()) | ||
| } | ||
|
|
||
| "ramalama" | "llama.cpp" => { |
There was a problem hiding this comment.
As you can see, they should be separated. However, the issue is that rig does not currently support Ramalama.
Rig provides an official solution for unsupported providers by exposing a trait that allows us to create a wrapper for new providers, making them compatible with rig.
See: link
What I found is that Ramalama (link) uses an OpenAI-compatible API, similar to llama.cpp (link).
Most local providers follow approximately the same approach.
Therefore, I used the llama client with Ramalama without needing to build a custom wrapper.
There was a problem hiding this comment.
The mcp_bridge.rs file was created specifically to solve an important issue: rig still uses an older version of rmcp (v0.16) for MCP communication, which caused a version conflict with pgmoneta_mcp.
To solve this issue, we will have to use the same version of rmcp as rig and rely on their version.
What was implemented is an Adapter that enables direct communication between the MCP server and rig without relying on rig’s rmcp.
There was a problem hiding this comment.
Yes, but we don't want this in our project. It need to be fixed upstream such that we don't have to do hacks like this
mashraf8
force-pushed
the
pgmoneta-llm-chat
branch
from
b75b2cb to
e2bcc1b
Compare
mashraf8
changed the title
|
@jesperpedersen, should I remove the |
|
@mashraf8 We have vLLM already. Yes, we need to refactor and remove the code provided by rig. Also, it is |
|
@mashraf8 Also, we need a rig.rs release that use rmcp 1.3+ so I'm converting this to draft |
|
@jesperpedersen I have already resolved the conflict issue by implementing a bridge in I think it would be better to keep this approach for now so we can keep up with the rapid updates and changes that occur continuously. At the same time, it would be better to focus on agents features such as RAG, multi-agent support, improving the interface, and other enhancements that directly impact the end user. |
|
@jesperpedersen Okay, I will proceed with adding the hook system and addressing any missing parts, and then open a new PR. |
…pp, Ramalama, vLLM), powered by rig
mashraf8
force-pushed
the
pgmoneta-llm-chat
branch
from
e2bcc1b to
b473eab
Compare
|
It will remain in draft status as is, as the update is not yet officially available on crates.io: I have temporarily used the latest version of rig-core from the rig repository, which includes the expected changes from the upcoming release, to continue working on the PR. |
|
@mashraf8 Yeah, we will wait a bit. You can try the client in |
|
@jesperpedersen a new version has been released with updates to rmcp. I will also apply the necessary changes and make some additional adjustments |
|
@mashraf8 I updated to rmcp 1.5 |
2 participants
@jesperpedersen @Jubilee101 Recently, I studied rig and how to integrate it. It is interesting, as it can indeed significantly simplify our work by removing the need to build a custom MCP client, now it already supports more than 20 providers. It also provides a hooks system that allows us to track events such as completion call, tool call, and more.
However, I have encountered some challenges, which will be mentioned as comments in the code.
Currently, implemented a CLI interactive chat that provides a smooth user experience and a clean user interface, with hooks helping achieve that, while also allowing users to change configuration directly within the chat.
it now supports providers such as Ollama, llama.cpp, and Ramalama, and we can easily add any provider supported by rig, for unsupported providers, rig also offers a solution.
The demo
Work is still ongoing on the documentation and improving the chat interface if needed.
Closes #80