[Bug][IPC] Python plugin IPC buffer is limited to 64 kb

[rohitgulia]

Search before asking

• I had searched in the issues and found no similar issues.

What happened

bufio.Scanner uses a 64KB default max token size. Python plugins that manage
large numbers of scopes (e.g. 100+ Harness pipelines) return a sync plan JSON
that exceeds this limit, causing silent truncation. This results in only a
subset of scopes appearing in project_mapping and being synced.

Increase the buffer to 5MB in both scanOutputPipe and scanErrorPipe 

eg of what to change: scanner.Buffer(make([]byte, 510241024), 510241024)

What do you expect to happen

increase scanner buffer from 64KB to 5MB for Python plugin IPC

bufio.Scanner uses a 64KB default max token size. Python plugins that manage
large numbers of scopes (e.g. 100+ Harness pipelines) return a sync plan JSON
that exceeds this limit, causing silent truncation. This results in only a
subset of scopes appearing in project_mapping and being synced.

Increase the buffer to 5MB in both scanOutputPipe and scanErrorPipe 

eg of what to change: scanner.Buffer(make([]byte, 510241024), 510241024)

How to reproduce

try syncing more than 100 pipelines via custom plugin

Anything else

No response

Version

v1.0.3-beta10

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[klesh]

Looking forward to your PR

[rohitgulia]

Looking forward to your PR

Hi @klesh , could you please add me as a contributor to this repo so that I can push my branch and create PR

[klesh]

Looking forward to your PR

Hi @klesh , could you please add me as a contributor to this repo so that I can push my branch and create PR

Please fork the repository, push your changes to your own fork, and then open a pull request from there.