Add LiveActivities metrics#1838
Conversation
| httpClient.sendAsync(request, HttpResponse.BodyHandlers.ofString()).whenComplete((response, err) => { | ||
| if (response == null) { | ||
| logger.error(s"Failed to create channel due to error ${err.getMessage}") | ||
| metrics.recordApnsNetworkError() |
There was a problem hiding this comment.
Even though sendAsync is used for the APNS call, I think CloudWatchClient.putMetricData is synchronous and blocks the whenComplete thread for handling the APNS response until CloudWatch responds. Have you considered Consider using CloudWatchAsyncClient
There was a problem hiding this comment.
Metrics are fire once after the broadcast to APNS goes out so there no impact on the service if metrics are fired synchronously.
For the channel creation its a very low volume metric count so would only fire once per invocation and not during a live event so again the extra time would be negligible.
For channel deletions, this is a chron job unrelated to the service performance so it doesn't need to be the fastest.
Overall it's a very low volume service and there is also a risk that the main lambda process terminates before the async cloud watch thread is finished and we loose metrics on a separate thread.
For these reasons I think firing synchronously is fine.
2 participants
What does this change?
Adds a metrics class with liveactivity lambda namespaces.
Adds metrics for:
APNS Responses to track 2xx/5xx/4xx and network erros
Our various "do not process" scenarios - these messages will still end up in the DLQ for investigation but we can see how frequently they occur over time with metrics.
This will allow us to easily spot problems talking to APNS, as well as the frequency overtime of outlier behaviour we are handling.
How has this change been tested?
Ran in CODE.
Metrics observed in Cloudwatch.
How can we measure success?
Have we considered potential risks?
Images
Accessibility