Skip to content

K8s shutdown#294

Open
SVyatoslavG wants to merge 4 commits into
yyyar:masterfrom
SVyatoslavG:k8s-shutdown
Open

K8s shutdown#294
SVyatoslavG wants to merge 4 commits into
yyyar:masterfrom
SVyatoslavG:k8s-shutdown

Conversation

@SVyatoslavG

Copy link
Copy Markdown

Added environment variable to support k8s graceful shutdown. It is not a graceful shutdown by definition.
It would be useful for running gobetween behind Google Cloud Loadbalancer. Kubernetes Engine sends syscall.SIGTERM and at the same time stop sending requests to the pod before shutdown. Because those actions are done at the same time race condition occurs and the process needs to sleep before exiting.
Please accept simple pull request.

@yyyar

yyyar commented Aug 18, 2020

Copy link
Copy Markdown
Owner

Kubernetes Engine sends syscall.SIGTERM and at the same time stop sending requests to the pod before shutdown. Because those actions are done at the same time race condition occurs and the process needs to sleep before exiting.

Could you please clarify why the process needs to sleep before exiting? So that we make sure gobetween won't shutdown while Kubernetes still sending requests to it? And also what value should GOBETWEEN_SHUTDOWN_TIME have - are there some recommendations for those who use Google Cloud Loadbalancer?
Thanks!

@SVyatoslavG

Copy link
Copy Markdown
Author

Sorry for the delay in answer.
Yes, Kubernetes still sending requests to pods that are about to get deleting. SIGTERM & routing of requests happen at the same time and cause race conditions.
I am using GOBETWEEN_SHUTDOWN_TIME 20s in my production environment. It is extremely hight for safety (theoretically GKE doesn't route any requests after 1s but for safety, I set 20s in case there are long-running requests).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants