Add /ready endpoint to DataPrepperServer for ALB health check readiness gating#6977
Conversation
| return; | ||
| } | ||
|
|
||
| final boolean allReady = pipelines.values().stream() |
There was a problem hiding this comment.
Might be good to just run one test yourself that uses a yaml with one true source like http / otel, and another "pipeline" source.
There was a problem hiding this comment.
Good point. I have tested locally with following config and added a temporary 30s sleep before source.start() to make the 503 window observable::
entry-pipeline:
source:
http:
port: 21890
ssl: false
sink:
- pipeline:
name: "downstream-pipeline"
downstream-pipeline:
source:
pipeline:
name: "entry-pipeline"
sink:
- stdout:
here is the ingestion result:
13:54:51 - HTTP 503 - {"ready":false,"reason":"pipelines not started: [entry-pipeline, downstream-pipeline]"}
13:54:52 - HTTP 503 - {"ready":false,"reason":"pipelines not started: [entry-pipeline, downstream-pipeline]"}
13:54:53 - HTTP 503 - {"ready":false,"reason":"pipelines not started: [entry-pipeline]"}
...
13:55:23 - HTTP 200 - {"ready":true}
The pipeline source (downstream-pipeline) starts instantly and drops off the 503 list within 1s. The http source (entry-pipeline) stays in 503 until source.start() completes and port 21890 is listening. /ready only returns 200 when all pipelines have their sources started.
| .filter(e -> !e.getValue().isSourceStarted()) | ||
| .map(Map.Entry::getKey) | ||
| .collect(Collectors.joining(", ")); | ||
| LOG.info("Readiness check: pipelines not ready: [{}]", notReady); |
There was a problem hiding this comment.
I would probably make this a DEBUG log as it will be a little noisy
There was a problem hiding this comment.
Good call! I changed it to LOG.debug. This will be hit every 30s by the ALB health check so INFO would be too noisy.
…all pipeline sources have started. This prevents ALB from routing traffic to targets before the data ingestion port is listening, eliminating 502 errors during scale-up events. Signed-off-by: Siqi Ding <dingdd@amazon.com>
0ef2937 to
25f6033
Compare
|
Looks good! |
3 participants
Description
Add a
/readyreadiness endpoint on the admin server (port 4900) that returns200 OKonly after all pipeline sources have started listening, and503 Service Unavailablewhile any pipeline source is still initializing.During auto-scaling events, the existing ALB health check on
/listpasses as soon as port 4900 starts, but the data ingestion port (e.g. 21890) hasn't started yet. The ALB routes traffic to the target, gets connection refused, and returns a 502 to the customer. This new endpoint enables the ALB health check to be pointed at/readyso targets are only marked healthy once they can actually serve traffic.Changes:
volatile boolean sourceStartedflag toPipeline.javathat flips aftersource.start()ReadinessHandlerimplementingGET /readythat checksisSourceStarted()on all pipelinesDataPrepperServerand Spring configurationIssues Resolved
NULL
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.