N.b., I'm happy to work up a PR for this, but before doing so I want to check your openness to the feature.
Problem: shovel is lightening fast. Often it's too fast during backfills and can overload my database's post processing capabilities (well less overload, more takes over my database). Also I blow out my credits very very quickly if I'm not extremely careful during backfill operations.
Current solution: I split my shovel into two instances, one that runs my "normal" operations and another that runs my backfills. I then manually turn my backfill on for a tranche during the day and watch my credits / database performance. When the credits start to get red for the day then I have to manually turn backfilling off and wait. Repeat the next day. FWIW, I've tried bringing down the batch_size and while that helps slightly in terms of not taking over my DB, what it really does is just end up being less efficient from a rate limit perspective with my node providers (e.g., it's more cost effective to get 100 blocks at a time than 20).
Proposed solution: currently there is the poll_duration setting for when the blocks are at head to keep rate limits within most providers requirements and to allow operators to tune to blocktime (no sense in pinging every second if the chain has a 12 second blocktime). It would be very useful to my operations if I could set a backfill_batch_duration or some other similar naming that would allow me to tune a time distance in between batches.
I'm happy to send over a PR but I want to check this would be a feature you'd be interested in before I work on it. If this isn't within the product philosophy you have for shovel that's perfectly fine. I can continue to backfill in the way I've become accustomed to.
N.b., I'm happy to work up a PR for this, but before doing so I want to check your openness to the feature.
Problem: shovel is lightening fast. Often it's too fast during backfills and can overload my database's post processing capabilities (well less overload, more takes over my database). Also I blow out my credits very very quickly if I'm not extremely careful during backfill operations.
Current solution: I split my shovel into two instances, one that runs my "normal" operations and another that runs my backfills. I then manually turn my backfill on for a tranche during the day and watch my credits / database performance. When the credits start to get red for the day then I have to manually turn backfilling off and wait. Repeat the next day. FWIW, I've tried bringing down the
batch_sizeand while that helps slightly in terms of not taking over my DB, what it really does is just end up being less efficient from a rate limit perspective with my node providers (e.g., it's more cost effective to get 100 blocks at a time than 20).Proposed solution: currently there is the
poll_durationsetting for when the blocks are at head to keep rate limits within most providers requirements and to allow operators to tune to blocktime (no sense in pinging every second if the chain has a 12 second blocktime). It would be very useful to my operations if I could set abackfill_batch_durationor some other similar naming that would allow me to tune a time distance in between batches.I'm happy to send over a PR but I want to check this would be a feature you'd be interested in before I work on it. If this isn't within the product philosophy you have for shovel that's perfectly fine. I can continue to backfill in the way I've become accustomed to.