The ShipLogEntries lambda in the mobile account experiences spikes in errors very frequently.
Upon inspection, these spikes were found to happen when a breaking news notification is sent. The amount of logging messages coming from the worker lambdas running in parallel overwhelms the limited number of shards of mobile-log-aggregation-PROD, which then leads to the lambda failing. It's important to note that when this happens, logs from all applications running in the account are dropped (including mapi), as they all share the same kinesis stream.
We tested switching the kinesis stream to on demand. This solved the problem, but at significant cost. The baseline price increased by a factor of 4 - from $180/month to $750/month. During the on demand test, the number of shards went up to 64, a significant departure from the original 3 shards it was running on.
The extreme spike in logs is difficult to accommodate. To stabilise the system and avoid the loss of logs, we'd recommend changing the lambda behaviour so it does not flood the stream with logs in the first place (maybe by using log sampling).
The ShipLogEntries lambda in the mobile account experiences spikes in errors very frequently.
Upon inspection, these spikes were found to happen when a breaking news notification is sent. The amount of logging messages coming from the worker lambdas running in parallel overwhelms the limited number of shards of mobile-log-aggregation-PROD, which then leads to the lambda failing. It's important to note that when this happens, logs from all applications running in the account are dropped (including mapi), as they all share the same kinesis stream.
We tested switching the kinesis stream to on demand. This solved the problem, but at significant cost. The baseline price increased by a factor of 4 - from $180/month to $750/month. During the on demand test, the number of shards went up to 64, a significant departure from the original 3 shards it was running on.
The extreme spike in logs is difficult to accommodate. To stabilise the system and avoid the loss of logs, we'd recommend changing the lambda behaviour so it does not flood the stream with logs in the first place (maybe by using log sampling).