JSON RPC responses are limited in size by size of pre-defined buffer

[oranjuice]

The workflow of using fossa as a JSON RPC server as per the example requires the creation of a buffer for the response to be filled into:
https://github.com/cesanta/fossa/blob/master/examples/json_rpc_server/json_rpc_server.c#L36

The problem here is that the size of a response can be really huge or really small depending on the method called or even the parameters passed to the same method. Pro-actively allocating a large buffer can be a needless waste of memory whereas allocating a small buffer can lead to buffer overflows.

Is there a way around this?

[cpq]

We can return a heap-allocated buffer and ask user to free it.
Do you have a suggestion?

[oranjuice]

But where we will the heap allocated buffer be returned from?
Also where should the user free it?

I was thinking maybe a char** initialized to NULL can be passed to the RPC handlers and the user allocates a char* to it as he wants and fills it. The problem is with freeing it. Maybe we could have a clean up function that the user supplies which takes care of cleaning up. Or the library could automatically clean it up. This isn't a very elegant solution though.

[cpq]

yeah, that approach stinks a little bit.
On the other hand, we've got a NS_HTTP_REQUEST message. That means that the whole request has been buffered, and we have full body available, and we know it's length: hm->body.len in the event handler. We can allocate the buffer on heap and pass it to the dispatch function. Will that work?

[oranjuice]

• The dispatch function should be the one deciding the size of the response and hence the buffer. Knowing the request body length might not be useful here.
• Even if we were to pass a heap allocated buffer to the dispatch function, who will free it?

PS: sorry if this was posted twice.

[cpq]

@oranjuice yes that's right.

Let's change the signature of dispatch function, from

int ns_rpc_dispatch(const char *buf, int, char *dst, int dst_len,
                    const char **methods, ns_rpc_handler_t *handlers);

to

int ns_rpc_dispatch(const char *buf, int, struct ns_connection *,
                    const char **methods, ns_rpc_handler_t *handlers);

The handler can print to the IO buffer directly.

[oranjuice]

That looks good!