CompactionVeryLargeST failed due to API Gateway LimitExceededException publishing query results to web socket

[patchwork01]

Description / Background

CompactionVeryLargeST failed because of a rate limit on API Gateway publishing query results to the web socket.

Steps to reproduce

1. Run CompactionVeryLargeST
2. Sometimes see error

Expected behaviour

The test should pass reliably.

Technical Notes / Implementation Details

We can review whether it's retrieving a reasonable number of rows through the web socket, and whether and how we could avoid the rate limit.

It doesn't look as though there are any relevant rate limits:
https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-execution-service-websocket-limits-table.html

The only thing there seems to be the payload size, and the error doesn't look like it's saying that's the problem.

Here's the documentation of the exception:

https://docs.aws.amazon.com/java/api/latest/software/amazon/awssdk/services/apigatewaymanagementapi/model/LimitExceededException.html

It says it can be that "the WebSocket client side buffer is full". Maybe it's a problem with our client side code, or something we need to handle on the server side.

Screenshots/Logs

Logs from leaf partition query processing lambda:

[main] query.lambda.SqsLeafPartitionQueryProcessor ERROR  - Error publishing results
java.io.UncheckedIOException: java.io.IOException: software.amazon.awssdk.services.apigatewaymanagementapi.model.LimitExceededException: Service returned error code LimitExceededException (Service: ApiGatewayManagementApi, Status Code: 429, Request ID: <request-id>) (SDK Attempt Count: 4)
at sleeper.query.runner.tracker.WebSocketQueryStatusReportDestination.send(WebSocketQueryStatusReportDestination.java:103)
at sleeper.query.runner.tracker.WebSocketQueryStatusReportDestination.queryCompleted(WebSocketQueryStatusReportDestination.java:77)
at sleeper.query.runner.tracker.WebSocketQueryStatusReportDestination.queryCompleted(WebSocketQueryStatusReportDestination.java:71)
at sleeper.query.runner.tracker.QueryStatusReportListeners.lambda$queryCompleted$5(QueryStatusReportListeners.java:109)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at sleeper.query.runner.tracker.QueryStatusReportListeners.queryCompleted(QueryStatusReportListeners.java:109)
at sleeper.query.core.model.QueryOrLeafPartitionQuery.reportCompleted(QueryOrLeafPartitionQuery.java:76)
at sleeper.query.lambda.SqsLeafPartitionQueryProcessor.publishResults(SqsLeafPartitionQueryProcessor.java:107)
at sleeper.query.lambda.SqsLeafPartitionQueryProcessor.processQuery(SqsLeafPartitionQueryProcessor.java:94)
at sleeper.query.lambda.SqsLeafPartitionQueryLambda.lambda$handleRequest$2(SqsLeafPartitionQueryLambda.java:102)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
at java.base/java.util.stream.Streams$StreamBuilderImpl.forEachRemaining(Streams.java:411)
at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:762)
at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:276)
at java.base/java.util.stream.ReferencePipeline$15$1.accept(ReferencePipeline.java:541)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
at sleeper.query.lambda.SqsLeafPartitionQueryLambda.handleRequest(SqsLeafPartitionQueryLambda.java:102)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at com.amazonaws.services.lambda.runtime.api.client.EventHandlerLoader$PojoMethodRequestHandler.handleRequest(EventHandlerLoader.java:741)
at com.amazonaws.services.lambda.runtime.api.client.EventHandlerLoader$PojoHandlerAsStreamHandler.handleRequest(EventHandlerLoader.java:658)
at com.amazonaws.services.lambda.runtime.api.client.EventHandlerLoader$2.call(EventHandlerLoader.java:604)
at com.amazonaws.services.lambda.runtime.api.client.AWSLambda.startRuntimeLoop(AWSLambda.java:317)
at com.amazonaws.services.lambda.runtime.api.client.AWSLambda.startRuntimeLoops(AWSLambda.java:267)
at com.amazonaws.services.lambda.runtime.api.client.AWSLambda.main(AWSLambda.java:224)
Caused by: java.io.IOException: software.amazon.awssdk.services.apigatewaymanagementapi.model.LimitExceededException: Service returned error code LimitExceededException (Service: ApiGatewayManagementApi, Status Code: 429, Request ID: <request-id>) (SDK Attempt Count: 4)
at sleeper.query.runner.websocket.ApiGatewayWebSocketOutput.sendString(ApiGatewayWebSocketOutput.java:94)
at sleeper.query.runner.tracker.WebSocketQueryStatusReportDestination.send(WebSocketQueryStatusReportDestination.java:101)
... 33 more
Caused by: software.amazon.awssdk.services.apigatewaymanagementapi.model.LimitExceededException: Service returned error code LimitExceededException (Service: ApiGatewayManagementApi, Status Code: 429, Request ID: <request-id>) (SDK Attempt Count: 4)
at software.amazon.awssdk.services.apigatewaymanagementapi.model.LimitExceededException$BuilderImpl.build(LimitExceededException.java:152)
at software.amazon.awssdk.services.apigatewaymanagementapi.model.LimitExceededException$BuilderImpl.build(LimitExceededException.java:100)
at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.retryPolicyDisallowedRetryException(RetryableStageHelper.java:180)
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:86)
at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:39)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:53)
at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:35)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:82)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:62)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:43)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:50)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:32)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37)
at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26)
at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:210)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:103)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:173)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:80)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:182)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:74)
at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:53)
at software.amazon.awssdk.services.apigatewaymanagementapi.DefaultApiGatewayManagementApiClient.postToConnection(DefaultApiGatewayManagementApiClient.java:315)
at software.amazon.awssdk.services.apigatewaymanagementapi.ApiGatewayManagementApiClient.postToConnection(ApiGatewayManagementApiClient.java:268)
at sleeper.query.runner.websocket.ApiGatewayWebSocketOutput.sendString(ApiGatewayWebSocketOutput.java:86)
... 34 more
Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 1 failure: Service returned error code LimitExceededException (Service: ApiGatewayManagementApi, Status Code: 429, Request ID: <request-id-attempt-1>)
Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 2 failure: Service returned error code LimitExceededException (Service: ApiGatewayManagementApi, Status Code: 429, Request ID: <request-id-attempt-2>)
Suppressed: software.amazon.awssdk.core.exception.SdkClientException: Request attempt 3 failure: Service returned error code LimitExceededException (Service: ApiGatewayManagementApi, Status Code: 429, Request ID: <request-id-attempt-3>)