feat(codecompanion): add format_action

[powerman]

Description

When tools is executed CodeCompanion does not show anything useful - all it says is "use_mcp_tool" has failed or succeeded. As "use_mcp_tool" is a meta-command, user do not see neither which tool was called, nor it args.

For successful execution it might be not critical and mostly just a curiosity to follow what LLM does.
But for failed runs it became a real UX issues, because in most cases failure happens because LLM uses wrong args, and now user needs to open a CodeCompanion debug output and try hard to find args used by LLM to spot an error and provide LLM with a hint how to fix it.

This PR adds support for a hook to let user format tool name and args to make them looks nice and useful.

Screenshots

Before this PR:

After this PR, using example configuration:

Checklist

• I've read the contributing guidelines and have adhered to them in this PR
• I've updated the README and/or relevant docs pages
• I've run make test to ensure all tests pass
• I've run make format to format the code
• I've run make docs to update the vimdoc pages

I've skipped make docs because it makes irrelevant changes - but it works ok for relevant changes.

[ravitemer]

@powerman Thank you for the PR. Sorry for the delay in response.

We have done a lot of refactoring at #169 with respect to tools. I think it's best we add this along with that. I'm thinking of replacing use_mcp_tool or access_mcp_resource with actual tool name on the server. And regarding showing the arguments in the chat, we already show the arguments nicely formatted in the confirmation window and I feel just the tool_name would be enough. What do you think? Does that work for you?

[powerman]

And regarding showing the arguments in the chat, we already show the arguments nicely formatted in the confirmation window and I feel just the tool_name would be enough.

I don't think so - both because of auto_approve for some/all tools, and because of re-reading chat history later. So, tool args (cleaned from too large args like text diff/new file content) in a chat looks useful to me. Can be optionally enabled, of course, or this (show args or not) can be controlled by generic args cleaning hook (which may return nothing if user don't wanna see any args).

We have done a lot of refactoring at #169 with respect to tools.

It turns out I'm now also working on tool calling issues with CodeCompanion/MCPHub:

• LLM sometimes provide tool args outside of tool_input
• LLM sometimes provide tool args as a JSON-string inside tool_input instead of JSON-object.

I was planning to open a discussion with you about it, but if you don't mind we can continue here.

I've just started, so some questions might be basic or invalid (sorry about that):

• Is CodeCompanion support real API to define available tools (I suppose it's a separate API param functions: [ ...JSON Schema...])? At a glance I don't see this.
• If it's not supported then I suppose main improvement in this field can be adding support for actual functions API in CodeCompanion. What do you think about this?
• I also noticed some possible issue with current MCPHub prompt. I'm not sure how critical they are (i.e. are they really affect LLM understanding of tool calling format), but at a glance they may worth fixing, just to clean up things at least (I can work on this if you think it makes sense):
  • Produced Markdown has some linting issues (multiple H1 headers etc.).
  • I think it's worth to minimize amount of unrelated context sent to LLM: e.g. in case of auto_toggle_mcp_servers = false or manually disabled MCPHub/toggle_mcp_server tool we can completely remove from prompt any mentions of disabled MCP servers at all.
  • Input Schema: part has Markdown formatting issues (no empty line before, a bit extra indent, no code block for schema).
  • It looks weird to me the schema itself is in Lua format (because of vim.inspect used to dump it) instead of JSON. Sure, LLM can read it, but there is a chance it'll work better if it will be in a proper markdown json codeblock and in JSON format.
  • Why are examples (for use_mcp_tool and access_mcp_resource) in a pseudocode instead of real format (JSON)? Isn't more realistic examples in a format model should actually use should work better?

Probably after #169 it will makes sense to describe all tools in API functions format instead of Markdown (and send this either in system prompt if CodeCompanion won't support actual functions API or append it as is into functions API value).

[ravitemer]

I don't think so - both because of auto_approve for some/all tools, and because of re-reading chat history later. So, tool args (cleaned from too large args like text diff/new file content) in a chat looks useful to me. Can be optionally enabled, of course, or this (show args or not) can be controlled by generic args cleaning hook (which may return nothing if user don't wanna see any args).

Yeah this makes sense. format_action seems to be ideal way to handle all cases. We provide basic like tool_name on server returned the following or something while allowing customization with format_action. Just thinking something like format_result might allow even more customization allowing users to decide the tool_name, arguments even the result of a tool call. This will override the show_result_in_chat which is currently all-or-none and allows to see result of some tools while ignore some others.

It turns out I'm now also working on tool calling issues with CodeCompanion/MCPHub:
LLM sometimes provide tool args outside of tool_input

Thanks and agree, some models do that like passing arguments in params, arguments, args etc based on the model. I haven't thought of handling these cases but that's an interesting idea. Really something to consider. However, with #169 you'd never run into this error as far as I tested. I'd like you to use the make_tools branch to see the result with proper function based tools. Mainly I use the @neovim tool group and it covers all the exploring and editing a project. I have updated the write_file and edit_file tools also with interactive diff, fuzzy matching etc

s CodeCompanion support real API to define available tools (I suppose it's a separate API param functions: [ ...JSON Schema...])? At a glance I don't see this.
If it's not supported then I suppose main improvement in this field can be adding support for actual functions API in CodeCompanion. What do you think about this?

I'm not sure I get this right but I'm thinking you are asking if we can create function tools from MCP server tools that work with codecompanion? If so, yes! We can just use the inputSchema of a tool as parameters for a function tool. This is code that handles converting each server tool into a codecompanion tool.

                    -- Add individual tool
                    tools[namespaced_tool_name] = {
                        id = "mcp_dynamic:" .. safe_name .. ":" .. tool_name,
                        description = tool.description,
                        hide_in_help_window = true,
                        visible = opts.show_server_tools_in_chat == true,
                        callback = {
                            name = namespaced_tool_name,
                            cmds = { create_individual_tool_handler(server.name, tool_name, namespaced_tool_name) },
                            output = core.create_output_handlers(namespaced_tool_name, true, opts),
                            schema = {
                                type = "function",
                                ["function"] = {
                                    name = namespaced_tool_name,
                                    description = tool.description,
                                    parameters = tool.inputSchema,
                                },
                            },
                        },
                    }

I majorly work with copilot adapter and so far I didn't run into any issues. Some old models might have an issue with tools having lengthy descriptions but I've not encountered in my testing. We can use system_prompt field if the need arises.

I also noticed some possible issue with current MCPHub prompt. I'm not sure how critical they are (i.e. are they really affect LLM understanding of tool calling format), but at a glance they may worth fixing, just to clean up things at least (I can work on this if you think it makes sense):
Produced Markdown has some linting issues (multiple H1 headers etc.).
I think it's worth to minimize amount of unrelated context sent to LLM: e.g. in case of auto_toggle_mcp_servers = false or manually disabled MCPHub/toggle_mcp_server tool we can completely remove from prompt any mentions of disabled MCP servers at all.
Input Schema: part has Markdown formatting issues (no empty line before, a bit extra indent, no code block for schema).
It looks weird to me the schema itself is in Lua format (because of vim.inspect used to dump it) instead of JSON. Sure, LLM can read it, but there is a chance it'll work better if it will be in a proper markdown json codeblock and in JSON format.

Agree with all these. Haven't focused much on them. Would really appreciate fixing these.

Why are examples (for use_mcp_tool and access_mcp_resource) in a pseudocode instead of real format (JSON)? Isn't more realistic examples in a format model should actually use should work better?

This was intentional because using actual JSON examples in the prompt surprisingly for some reason didn't work as expected. It was a hit and miss for different models. I thought that if we are expecting structured output, JSON examples work best but not for function calling. There seems to be some different internal schema these models are trained on and they are getting confused between the JSON examples and their internal examples I think.

[powerman]

I'd like you to use the make_tools branch

Yeah, that's the plan for today. 😄

I'm not sure I get this right but I'm thinking you are asking if we can create function tools from MCP server tools that work with codecompanion?

No, I was talking about sending available functions to API in a separate "functions" key instead of usual system prompt in a "messages" key. Check this example with both keys: https://gist.github.com/hansvdam/8b9269390e16fa0bf394d7656bec1ea5 (that's just a random gist I just found, can't guarantee it's valid, but at a glance it is valid).

Moving this way will result in removing markdown system prompt with tools at all. Or maybe transform it to contain same JSON as value of "functions" API param to use for providers which does not support "functions" param in API (if such providers still exists).

[powerman]

Mainly I use the @neovim tool group and it covers all the exploring and editing a project.

Actually I've completely disabled both MCPHub and Neovim groups (and also I don't use any of CodeCompanion tools). I'm using 3rd-party MCP servers like filesystem and tavily-mcp from Marketplace and some external ones like https://context7.com/ (BTW it's worth adding to the Marketplace!) and https://github.com/sonirico/mcp-shell.

The rationale for this:

• Security. I consider toggle_mcp_server a security hole because I've disabled some MCP tools for a reason and don't want LLM to be able to re-enable them. (I do see some cool use cases enabled with this tool of course, but I personally bother much more about security and have to avoid such use cases even if they're cool. 😞)
• I prefer to limit shell command execution in some ways - at very minimum is to add a timeout and limit output size. These (and some extra security-related) features are provided by mcp-shell but not by CodeCompanion's cmd_runner or MCPHub's Neovim/execute_command.
• I'd like to use LLM in "agent mode" (like in Cursor/Zed/Windsurf). This means it should be able in one go to modify several files, run tests, fix issues, and only then report "it's done". Maybe I'm just doing something wrong, but I was unable to find out a way to use neither CodeCompanion's tools nor MCPHub's Neovim tools this way: it looks like these tools are tries to change Neovim buffers (always or only if there is an open buffer for a modified file) and ask user to manually "approve" each change - and that's a problem because I both don't wanna review and approve each invalid intermediate change (most of LLM changes are invalid until it runs tests and fixes all issues) and I wanna let LLM run tests which means LLM changes should be saved into files instead of stored in modified Neovim buffers. So, I've disabled all such file editing tools (to ensure LLM won't use them by mistake) and installed MCP filesystem server instead.

[ravitemer]

No, I was talking about sending available functions to API in a separate "functions" key instead of usual system prompt in a "messages" key. Check this example with both keys: https://gist.github.com/hansvdam/8b9269390e16fa0bf394d7656bec1ea5 (that's just a random gist I just found, can't guarantee it's valid, but at a glance it is valid).

Yeah, make_tools converts each MCP tool to a CodeCompanion tool and CodeCompanion passes these tools in the functions key depending on the adapter. We don't use system_prompt anymore except for the current @mcp group which uses the current approach of providing two function tools use_mcp_tool and access_mcp_resource which will be sent via functions key and the running servers via system prompt in messages. With make_tools when you add a server or a tool in the server in the chat @neovim or @neovim__list_directory messages table is not changed, we add tools to the functions key.

Actually I've completely disabled both MCPHub and Neovim groups (and also I don't use any of CodeCompanion tools). I'm using 3rd-party MCP servers like filesystem and tavily-mcp from Marketplace and some external ones like https://context7.com/ (BTW it's worth adding to the Marketplace!) and https://github.com/sonirico/mcp-shell.

We now manage our own registry https://github.com/ravitemer/mcp-registry with static configs. See #192

I'd like to use LLM in "agent mode" (like in Cursor/Zed/Windsurf). This means it should be able in one go to modify several files, run tests, fix issues, and only then report "it's done". Maybe I'm just doing something wrong, but I was unable to find out a way to use neither CodeCompanion's tools nor MCPHub's Neovim tools this way: it looks like these tools are tries to change Neovim buffers (always or only if there is an open buffer for a modified file) and ask user to manually "approve" each change - and that's a problem because I both don't wanna review and approve each invalid intermediate change (most of LLM changes are invalid until it runs tests and fixes all issues) and I wanna let LLM run tests which means LLM changes should be saved into files instead of stored in modified Neovim buffers. So, I've disabled all such file editing tools (to ensure LLM won't use them by mistake) and installed MCP filesystem server instead.

Make sure to auto approve the edit_file or write_file in the :MCPHub with a.

[powerman]

• It looks weird to me the schema itself is in Lua format (because of vim.inspect used to dump it) instead of JSON.

@ravitemer It turns out Neovim has no pretty-print for JSON. I still think using JSON here is better, but I'm not sure which option to choose:

• Do not pretty-print, output whole JSON in a single line, as returned by vim.json.encode or vim.fn.json_encode. The reader is an LLM, so it should be just fine.
• Add extra file (or a dir) with 3rd-party implementation like https://gist.github.com/smjonas/d5faacaa01cfe8d80075279e221efc83 (MIT license) or https://github.com/smjonas/snippet-converter.nvim/blob/main/lua/snippet_converter/utils/json_utils.lua (MPL-2.0 license).

[ravitemer]

We have a M.pretty_json function in utils/init.lua which is how we render the json in the :MCPHub

[powerman]

We have a M.pretty_json function in utils/init.lua which is how we render the json in the :MCPHub

Cool! I'm going to make a PR with switching Schema to JSON, and then look how to adapt current PR (format_action) for the make-tools branch and probably will open another one targeted to make-tools branch.

[ravitemer]

Thanks for the new PR. Closing this