Skip to content

fix: set header snippets for websocket upgrade on nginx-ingress#7881

Merged
agoose77 merged 2 commits into
2i2c-org:mainfrom
agoose77:fix-nginx-ingress
Mar 9, 2026
Merged

fix: set header snippets for websocket upgrade on nginx-ingress#7881
agoose77 merged 2 commits into
2i2c-org:mainfrom
agoose77:fix-nginx-ingress

Conversation

@agoose77

@agoose77 agoose77 commented Mar 9, 2026

Copy link
Copy Markdown
Contributor

@ktyle reported that kernels and terminals weren't working properly on the Pythia BinderHub. He also saw 502 errors. Both of these, I reproduced.

We recently upgraded that hub to use nginx-ingress, and it turns out that our current configuration does not support websocket upgrades.

This PR sets the location snippet required to support that. There may be other configuration we need to tweak here (like buffering, timeouts, etc).

It also sets a TTL on DNS replies to match the old ingress defaults.

@github-actions

github-actions Bot commented Mar 9, 2026

Copy link
Copy Markdown
Contributor

Merging this PR will trigger the following deployment actions.

Support deployments

No support upgrades will be triggered

Staging deployments

Cloud Provider Cluster Name Hub Name Reason for Redeploy
aws victor staging Core infrastructure has been modified
aws smithsonian staging Core infrastructure has been modified
aws earthscope staging Core infrastructure has been modified
aws nasa-veda staging Core infrastructure has been modified
aws nasa-ghg-hub staging Core infrastructure has been modified
aws jupyter-health staging Core infrastructure has been modified
aws nasa-cryo staging Core infrastructure has been modified
gcp 2i2c staging Core infrastructure has been modified
gcp 2i2c dask-staging Core infrastructure has been modified
kubeconfig utoronto staging Core infrastructure has been modified
kubeconfig utoronto r-staging Core infrastructure has been modified
gcp cloudbank staging Core infrastructure has been modified
aws projectpythia staging Core infrastructure has been modified
aws aimatx-2i2c-hub staging Core infrastructure has been modified
aws nmfs-openscapes staging Core infrastructure has been modified
kubeconfig 2i2c-jetstream2 staging Core infrastructure has been modified
gcp leap staging Core infrastructure has been modified
aws openscapeshub staging Core infrastructure has been modified
gcp hhmi staging Core infrastructure has been modified
aws disasters staging Core infrastructure has been modified
aws temple staging Core infrastructure has been modified
aws reflective staging Core infrastructure has been modified
aws opensci staging Core infrastructure has been modified
aws maap staging Core infrastructure has been modified
gcp 2i2c-uk staging Core infrastructure has been modified
aws berkeley-geojupyter staging Core infrastructure has been modified
aws ucmerced staging Core infrastructure has been modified
aws strudel staging Core infrastructure has been modified
gcp awi-ciroh staging Core infrastructure has been modified
aws 2i2c-aws-us staging Core infrastructure has been modified
aws bnext-bio staging Core infrastructure has been modified

Production deployments

Cloud Provider Cluster Name Hub Name Reason for Redeploy
aws victor prod Core infrastructure has been modified
aws smithsonian prod Core infrastructure has been modified
aws earthscope prod Core infrastructure has been modified
aws earthscope binder Core infrastructure has been modified
aws nasa-veda prod Core infrastructure has been modified
aws nasa-veda binder Core infrastructure has been modified
aws nasa-ghg-hub prod Core infrastructure has been modified
aws nasa-ghg-hub binder Core infrastructure has been modified
aws jupyter-health prod Core infrastructure has been modified
aws nasa-cryo prod Core infrastructure has been modified
kubeconfig projectpythia-binder binderhub Core infrastructure has been modified
gcp 2i2c mtu Core infrastructure has been modified
kubeconfig utoronto prod Core infrastructure has been modified
kubeconfig utoronto r-prod Core infrastructure has been modified
kubeconfig utoronto highmem Core infrastructure has been modified
gcp cloudbank ahs Core infrastructure has been modified
gcp cloudbank authoring Core infrastructure has been modified
gcp cloudbank bcc Core infrastructure has been modified
gcp cloudbank bmcc Core infrastructure has been modified
gcp cloudbank chaffey Core infrastructure has been modified
gcp cloudbank ccsf Core infrastructure has been modified
gcp cloudbank chabot Core infrastructure has been modified
gcp cloudbank chicagostate Core infrastructure has been modified
gcp cloudbank cmu Core infrastructure has been modified
gcp cloudbank csm Core infrastructure has been modified
gcp cloudbank csum Core infrastructure has been modified
gcp cloudbank deanza Core infrastructure has been modified
gcp cloudbank demo Core infrastructure has been modified
gcp cloudbank dvc Core infrastructure has been modified
gcp cloudbank elac Core infrastructure has been modified
gcp cloudbank elcamino Core infrastructure has been modified
gcp cloudbank evc Core infrastructure has been modified
gcp cloudbank etsu Core infrastructure has been modified
gcp cloudbank fresno Core infrastructure has been modified
gcp cloudbank foothill Core infrastructure has been modified
gcp cloudbank glendale Core infrastructure has been modified
gcp cloudbank golden Core infrastructure has been modified
gcp cloudbank gwu Core infrastructure has been modified
gcp cloudbank gpu-demo Core infrastructure has been modified
gcp cloudbank high Core infrastructure has been modified
gcp cloudbank humboldt Core infrastructure has been modified
gcp cloudbank kean Core infrastructure has been modified
gcp cloudbank lacc Core infrastructure has been modified
gcp cloudbank lahc Core infrastructure has been modified
gcp cloudbank laney Core infrastructure has been modified
gcp cloudbank lavc Core infrastructure has been modified
gcp cloudbank lbcc Core infrastructure has been modified
gcp cloudbank mendocino Core infrastructure has been modified
gcp cloudbank merced Core infrastructure has been modified
gcp cloudbank merritt Core infrastructure has been modified
gcp cloudbank mmc Core infrastructure has been modified
gcp cloudbank miracosta Core infrastructure has been modified
gcp cloudbank mission Core infrastructure has been modified
gcp cloudbank moreno Core infrastructure has been modified
gcp cloudbank norco Core infrastructure has been modified
gcp cloudbank ocu Core infrastructure has been modified
gcp cloudbank palomar Core infrastructure has been modified
gcp cloudbank pasadena Core infrastructure has been modified
gcp cloudbank redwoods Core infrastructure has been modified
gcp cloudbank reedley Core infrastructure has been modified
gcp cloudbank riohondo Core infrastructure has been modified
gcp cloudbank saddleback Core infrastructure has been modified
gcp cloudbank santiago Core infrastructure has been modified
gcp cloudbank sbcc Core infrastructure has been modified
gcp cloudbank sbcc-dev Core infrastructure has been modified
gcp cloudbank sierra Core infrastructure has been modified
gcp cloudbank sjcc Core infrastructure has been modified
gcp cloudbank sjsu Core infrastructure has been modified
gcp cloudbank skyline Core infrastructure has been modified
gcp cloudbank sou Core infrastructure has been modified
gcp cloudbank stanford Core infrastructure has been modified
gcp cloudbank spelman Core infrastructure has been modified
gcp cloudbank srjc Core infrastructure has been modified
gcp cloudbank tuskegee Core infrastructure has been modified
gcp cloudbank ucsc Core infrastructure has been modified
gcp cloudbank uchicago Core infrastructure has been modified
gcp cloudbank umd Core infrastructure has been modified
gcp cloudbank und Core infrastructure has been modified
gcp cloudbank virginia Core infrastructure has been modified
gcp cloudbank wlac Core infrastructure has been modified
aws projectpythia prod Core infrastructure has been modified
aws projectpythia pythia-binder Core infrastructure has been modified
aws aimatx-2i2c-hub prod Core infrastructure has been modified
aws nmfs-openscapes prod Core infrastructure has been modified
aws nmfs-openscapes workshop Core infrastructure has been modified
aws nmfs-openscapes noaa-only Core infrastructure has been modified
gcp dubois ephemeral Core infrastructure has been modified
gcp leap prod Core infrastructure has been modified
gcp leap public Core infrastructure has been modified
aws openscapeshub prod Core infrastructure has been modified
aws openscapeshub workshop Core infrastructure has been modified
gcp hhmi spyglass Core infrastructure has been modified
gcp hhmi binder Core infrastructure has been modified
aws disasters prod Core infrastructure has been modified
aws temple prod Core infrastructure has been modified
aws temple advanced Core infrastructure has been modified
aws temple research Core infrastructure has been modified
aws reflective prod Core infrastructure has been modified
aws reflective workshop Core infrastructure has been modified
aws opensci sciencecore Core infrastructure has been modified
aws opensci climaterisk Core infrastructure has been modified
aws opensci small-binder Core infrastructure has been modified
aws opensci big-binder Core infrastructure has been modified
aws maap prod Core infrastructure has been modified
gcp 2i2c-uk lis Core infrastructure has been modified
aws berkeley-geojupyter prod Core infrastructure has been modified
aws ucmerced prod Core infrastructure has been modified
aws strudel prod Core infrastructure has been modified
aws strudel workshop Core infrastructure has been modified
gcp awi-ciroh prod Core infrastructure has been modified
aws 2i2c-aws-us showcase Core infrastructure has been modified
aws bnext-bio prod Core infrastructure has been modified

@agoose77 agoose77 requested a review from yuvipanda March 9, 2026 20:08
Comment on lines +16 to +17
nginx.org/server-snippets: |
resolver_timeout 30s;

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like NGINX caches the DNS responses by default, whereas the existing ingress-nginx uses the /etc/resolve to set a resolver with 30s TTL.

@agoose77

agoose77 commented Mar 9, 2026

Copy link
Copy Markdown
Contributor Author

@yuvipanda I pinged you for review. This feels a bit like whack-a-mole — I wonder whether we should do a more systematic comparison of the resolved nginx configurations (I've been using them to guess what's missing).

@agoose77 agoose77 moved this to In progress in Product and Services Mar 9, 2026
@agoose77 agoose77 self-assigned this Mar 9, 2026
@yuvipanda

Copy link
Copy Markdown
Member

@agoose77 whackamole is fine for now. Let's do this and see if this solves the issue? I think the resolved nginx config will have too much accidental differences

@agoose77

agoose77 commented Mar 9, 2026

Copy link
Copy Markdown
Contributor Author

? I think the resolved nginx config will have too much accidental differences

Yeah, they're fairly different!

@agoose77 agoose77 merged commit 7687203 into 2i2c-org:main Mar 9, 2026
41 checks passed
@github-project-automation github-project-automation Bot moved this from In progress to Done in Product and Services Mar 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants