Post processing

[vkolotov]

Hey @timesler, the new just_bash makes it really efficient, I can see 98% of improvements in terms of token usage in claude code:

The two layers do different jobs:
  - mcp-compressor compresses 59 tool schemas (~23,600 tok) into one compressed description (~1,430 tok) — 94% schema saving
  - Bash filtering cuts result tokens from 222K down to 1.8K — 99.2% result saving

  Combined: 35× fewer tokens vs original uncompressed MCP tools.

  In dollars at Sonnet's input price ($3/MTok, with caching on schemas):
  - Uncompressed: ~$0.78 per workflow run
  - Compressed + filtered: ~$0.01
  - ~$0.77 saved per run — or roughly 98.5%

The resulting md files for my agents have a bunch of jq etc piping. Essentially what they do is two things:

• Filter rows
• Filter columns

Some of tools I use have json output data format, some (weirdly) yaml and csv.

I wonder if it makes sense to implement automatic data format convesion in this lib, so that (for ex) agents always use json and they can easily and in a unified way use jq to access fiels they only need.

Handling CSV and yaml is very fragile and difficult to support.

[timesler]

@vkolotov this lib already does some data format conversion, but it's specifically designed to prevent JSON outputs because they consume so many tokens. The default in cli-mode or just-bash mode is to convert any JSON outputs from tools into TOON format (reduces tokens by about 40%)

However, that auto-conversion is definitely a problem for piping stuff, so I have just implemented a new feature for both cli-mode and just-bash that auto-detects if the output will be piped or redirected and disables the toonification.

In terms of detecting other formats and converting them, that sounds like a reasonable thing to do for both cli-mode and just-bash, and then it could automatically flow through the pipe/redirect detection and TOON conversion too

So we would have something like this:

MCP Tool ---------------------> Agent
          # When being called directly
          JSON -> JSON -> TOON
          CSV  -> JSON -> TOON
          YAML -> JSON -> TOON
          
          # When being piped or redirected (e.g., to a file)
          JSON -> JSON -> JSON
          CSV  -> JSON -> JSON
          YAML -> JSON -> JSON