Bulk ingestion is extremely slow.

[RaulRoPer]

I have been working with Trino for years, and this is a problem specific to Trino. Each insert takes a long time, and its Bulk Ingestion implementation is executing one insert per row, which in this case is inefficient and extremely slow.

The only way I have achieved acceptable performance is by batching rows for each insert, so I suggest you adopt that strategy and insert multiple rows, adding a customisable parameter ‘number_batches’ such that: INSERT INTO ... VALUES ((1,7,3000),(1,8,3500),(1,9,3900))

[lidavidm]

CC @Mandukhai-Alimaa

This is a common optimization we'll have to deal with for databases like Trino that don't offer a better path (e.g. I think pandas does it too), so we should implement it in sqlwrapper

[RaulRoPer]

Trino has a default parameter setting of query.max-length=1000000 (config.properties) characters ≈1 MB.
Very large SQL may exceed memory :

Perhaps it would be much more stable to set a parameter to fix the maximum batch size, "max_payload=1MB" instead of number_batches.