I use Django + Celery + Beat to manage many recurring background tasks, which have been working okay apart from one crucial problem.
Beat seems to be sending the messages into SQS very slowly, about 100/minute.
Every Sunday I have a sendout to about 16k users, and they're all booked for 6.30pm. Beat starts picking it up at the expected time, and I would expect a huge spike in messages coming into SQS at that time, but it takes its time, and I can see on the logs that the "Sending tasks x..." goes on for a few hours.
I have autoscaling on for my Celery workers, but because the number of messages doesn't really ever spike, the workers don't really scale until later, when the messages start backing up a bit.
I'm really puzzled by this behaviour, anyone there know what I could be missing?
I'm running celery with, some cron tab tasks but this one task in specific is a PeriodicTask
celery_beat: celery -A appname beat --loglevel=INFO
Thanks!
I use Django + Celery + Beat to manage many recurring background tasks, which have been working okay apart from one crucial problem.
Beat seems to be sending the messages into SQS very slowly, about 100/minute.
Every Sunday I have a sendout to about 16k users, and they're all booked for 6.30pm. Beat starts picking it up at the expected time, and I would expect a huge spike in messages coming into SQS at that time, but it takes its time, and I can see on the logs that the "Sending tasks x..." goes on for a few hours.
I have autoscaling on for my Celery workers, but because the number of messages doesn't really ever spike, the workers don't really scale until later, when the messages start backing up a bit.
I'm really puzzled by this behaviour, anyone there know what I could be missing?
I'm running celery with, some cron tab tasks but this one task in specific is a PeriodicTask
celery_beat: celery -A appname beat --loglevel=INFOThanks!