Cap'n Proto RPC ByteStream uses more CPU than gRPC bidirectional streaming

[shizhx]

Same condition:

• bandwidth: 1Gbps
• block size : 1500

toplogy:
iperf3 client -> proxy-bench client -> proxy-bench server -> iperf3 server

capnp rpc:

gRPC:

Core Code here:
https://github.com/shizhx/proxy-bench/blob/096738d67bd80a936207c2bdf1b261da898a0ae9/capnpnet/capnp_net.go#L244-L274

https://github.com/shizhx/proxy-bench/blob/096738d67bd80a936207c2bdf1b261da898a0ae9/capnpnet/capnp_net.go#L187-L217

Why is that? Shouldn't Cap'n Proto have less overhead since it doesn't involve serialization and deserialization?

This is my first time working with Cap'n Proto RPC. If there are any issues with the code, please feel free to point them out. Thank you!

[matheusd]

The current design and implementation of go-capnp isn't particularly geared towards high performance. I have designed a new implementation from the ground up to try and address these issues, but it's still early days (implementation https://github.com/matheusd/mdcapnp and an RPC perf comparison https://github.com/matheusd/gorpcbench).

I'll see if I can try out your benchmark for comparison.

[shizhx]

@matheusd I will try it later, I am looking for a high-performance multiplexing solution (including an RPC library) for network proxy/tunneling, so I am focusing on testing the performance of bidirectional streams.

[matheusd]

You'll have a hard time trying it out :P

It's still at a pretty experimental stage (I don't even have a compiler hooked up yet, I just hand-write all capnp structures and interfaces).

What/how are you measuring that %cpu in the screenshots?

That pprof output is for which execution (client or server)?

[shizhx]

@matheusd I use pidstat -r -u -p <pid> 2 15 command to measure average CPU usage, and the pprof was captured from client side(sending side).

[matheusd]

Ok, thanks. I'll see if I can write an implementation of this bench that uses my experiment.

[matheusd]

Ok, here are the comparisons on my machine. Code is for matheusd/proxy-bench@13cac60. I had to add some missing features on my experimental implementation (and they're totally WIP, so not reliable for actual production, but should be reasonable for the benchmarking experiment).

Hopefully I've replicated your benchmarking harness close enough.

gRPC

go-CapNProto

md-CapNProto

[shizhx]

@matheusd Great results. Could you compare it with native TCP as well? My biggest hope is that Cap’n Proto RPC performs about the same as TCP.

[matheusd]

Given these were run in different days, I can't guarantee my setup was exactly the same between them, but it should be close enough for your purposes.

Edit: I've re-ran the mdcapnp version and it's basically the same results as the prior one (to within 1pp).

tcp