Late marshalling, streaming

[len-ro]

Hello,

I am atempting the migrating of a large project from jgroups 3 to jgroups 5 and I have found the doc about late marshalling. This sounds great as we had to implement some kind of chunking mechanism in order to send large data without blowing the memory out. What would be the recommend method of sending large data in jgroups 5? Can I hope to wrap an InputStream in a serializable envelope and expect the chunking to be done at protocol level?

Regards,

[belaban]

You can use FRAG4 to fragment / unfragment large messages to/from smaller messages. This works with ObjectMessages, too.

Alternatively, you could do something like StreamingStateTransfer [1] at the application level (take a look at STATE or STATE_SOCK).

[1] https://github.com/belaban/JGroups/blob/master/src/org/jgroups/protocols/pbcast/StreamingStateTransfer.java

[onestardao]

very familiar territory — large payload transfer over protocol layers always ends up hitting three invisible landmines:

[No.11] serialization envelope mismatch (i.e., you wrap, protocol fragments, and decoding fails mid-boundary)
[No.14] memory blowup from eager chunk accumulation pre-stream
[No.15] chunk-size fixed upstream, but downstream expects adaptive flow control or codec alignment

jgroups does offer FRAG and StreamingStateTransfer, but unless your upstream can negotiate stream-aware chunks (and not just byte slices), the protocol-level fragmenting ends up being blind to semantic boundaries.

we’ve had to build something that watches chunk formation + transfer + reassembly as a coherent semantic unit — if you want, i can share how we route around [No.11] through [No.15] and keep memory safe without rewriting protocol code.

let me know.

[len-ro]

Thanks,

Hit no. 14 many times. I was hopping to be able to remove most customized code and ease further migrations in the future, we had a lot of too-deep changes which took time to migrate. For instance I will drop all custom encryption and rely on the native one. Was kind of hopping to be able to just provide an input stream and let the protocol do everything :) (chunking, package ordering, retransmissions, etc.). Will be grateful for any knowledge :)

Regards,

[onestardao]

got it

and yeah, No.14 is nasty, especially when upstream gets eager and downstream is expecting backpressure-aware reassembly.

we ended up building a layer that treats chunking, ordering, and retransmission as a single semantic stream unit — avoids memory blowup, lets us handle edge-alignment (No.11), and keeps retries from silently duplicating mid-stream payloads (No.15). kind of sits orthogonal to the protocol layer.

if you're curious, happy to share the full mapping ~ we tracked these under ProblemMap here:
https://github.com/onestardao/WFGY/blob/main/ProblemMap/README.md

the whole system is MIT licensed ~ no rewrite required, just drop in where the stream gets split.