Why does block size is limited at 16Mb ?

[chambo-e]

Hello,

Just saw that blockSize is limited at 16MB while trying to set one at 32MB

juicefs/cmd/format.go

Line 39 in 3190a02

func fixObjectSize(s int) int {

I wasn't able to find a mention of this limit in the docs, was wondering if there was any ?

Thanks :)

[davies]

Block is the unit of PUT/GET and also cached in disk. For bigger block size, it will consume more memory and easy to fail, and have not done much tests with them.

Why do you prefer bigger block size?

[chambo-e]

I use an S3 provider which recommend to keep < 500K objects in a bucket, otherwise performance might suffer

Having bigger blocks means I could stay below 500k :)

[xiaogaozi]

Maybe you should try --shards option in juicefs format command.

[chambo-e]

Ooh, I missed that option, thanks a lot, works like a charm !

[jheredianet]

Please could you give an example of using this?
Thanks in advance.