Running jupyterhub

[lauri-codes]

I wanted to try and run the jupyterhub part of the dev setup, and was following the README. When I first run uv run poe hub after doing the setup, I get an error:

Failed to find proxy ['configurable-http-proxy']
    The proxy can be installed with `npm install -g configurable-http-proxy`.To install `npm`, install nodejs which includes `npm`.If you see an `EACCES` error or permissions error, refer to the `npm` documentation on How To Prevent Permissions Errors.
[C 2025-03-12 14:27:20.634 JupyterHub app:3139] Failed to start proxy
    Traceback (most recent call last):
      File "/home/lauri/repositories/nomad-distro-dev/.venv/lib/python3.12/site-packages/jupyterhub/app.py", line 3137, in start
        await self.proxy.start()
      File "/home/lauri/repositories/nomad-distro-dev/.venv/lib/python3.12/site-packages/jupyterhub/proxy.py", line 754, in start
        self.proxy_process = Popen(
                             ^^^^^^
      File "/home/lauri/.local/share/uv/python/cpython-3.12.6-linux-x86_64-gnu/lib/python3.12/subprocess.py", line 1026, in __init__
        self._execute_child(args, executable, preexec_fn, close_fds,
      File "/home/lauri/.local/share/uv/python/cpython-3.12.6-linux-x86_64-gnu/lib/python3.12/subprocess.py", line 1955, in _execute_child
        raise child_exception_type(errno_num, err_msg, err_filename)
    FileNotFoundError: [Errno 2] No such file or directory: 'configurable-http-proxy'

This could be fixed by installing that npm package: npm install -g configurable-http-proxy. Did I miss some step in the install, or should we document this step in the README? Or can we install that package as part of the setup? I did not look much further who is trying to run this proxy (comes from somewhere underneath nomad admin run hub).

[lauri-codes]

Also related to this: after installing the proxy, I still am not able to launch tools for some reason. I checked the following:

• I see that hub itself is running fine: I can access e.g. http://localhost:9000/fairdi/nomad/latest/north/hub/home
• On the first run I see that Jupyter starts downloading the images, which is fine and I just wait for them to be pulled
• I have checked that I belong to the docker group (getent group | grep docker says docker:x:998:lauri, my user is in group 998)
• I have run sudo chown -R 1000 .volumes

Still, when trying to launch e.g. jupyter notebook I get the following logs in uv run poe hub:

[I 2025-03-13 08:41:09.487 JupyterHub provider:659] Creating oauth client jupyterhub-user-lauri%20himanen-jupyter
[I 2025-03-13 08:41:09.507 JupyterHub dockerspawner:1033] Container 'nomad_oasis_north-lauri-20himanen--jupyter' is gone
[I 2025-03-13 08:41:09.539 JupyterHub dockerspawner:1311] Created container nomad_oasis_north-lauri-20himanen--jupyter (id: c6a838c) from image gitlab-registry.mpcdf.mpg.de/nomad-lab/nomad-distro/jupyter:develop
[I 2025-03-13 08:41:09.539 JupyterHub dockerspawner:1335] Starting container nomad_oasis_north-lauri-20himanen--jupyter (id: c6a838c)
[W 2025-03-13 08:41:19.496 JupyterHub base:1122] User lauri himanen:jupyter is slow to become responsive (timeout=10)
[I 2025-03-13 08:41:19.496 JupyterHub log:191] 202 POST /fairdi/nomad/latest/north/hub/api/users/lauri%20himanen/servers/jupyter (nomad-service@::ffff:127.0.0.1) 10035.47ms
[I 2025-03-13 08:42:12.725 JupyterHub dockerspawner:1033] Container 'nomad_oasis_north-lauri-20himanen--jupyter' is gone
[W 2025-03-13 08:42:12.725 JupyterHub dockerspawner:1008] Container not found: nomad_oasis_north-lauri-20himanen--jupyter

I would suspect that this is somehow related to the fact that the hub service is not able to run docker containers. @fekad: Maybe you could also try this and see if you can figure out what is wrong? Probably I missed something minor in setting things up, but then we need to just put some missing steps into the README.

[fekad]

1. I agree, the npm install -g configurable-http-proxy should be mentioned in the README.

2. The issue here is that your hub uses the fairdi_nomad_prod instead of the fairdi_nomad_test as it is defined in the nomad.yaml file. You should get Creating oauth client jupyterhub-user-test-jupyter instead of Creating oauth client jupyterhub-user-lauri%20himanen-jupyter.
   It looks like the poe task changes the CWD so it doesn't use the same nomad.yaml file for the configuration. Here is a fix:

   [tool.poe.tasks.hub]
   cmd = "nomad admin run hub"
   cwd = "packages/nomad-FAIR"
   env = { NOMAD_CONFIG = "../../nomad.yaml" }
   

[DanielYang59]

I might be having the same issue, and I added the NOMAD_CONFIG to nomad.yaml under nomad-distro-dev but it doesn't seem to fix.

[tool.poe.tasks.hub]
cmd = "nomad admin run hub"
cwd = "packages/nomad-FAIR"
+ env = { NOMAD_CONFIG = "/home/yang/nomad/nomad-distro-dev/nomad.yaml" }

[I 2026-03-31 16:53:10.492 JupyterHub provider:660] Creating oauth client jupyterhub-user-haoyutest

On my machine the hub URL cannot be resolved correctly, passing an explicit hub_connect_url to point to the docker bridge gateway (check via docker network inspect bridge) fixed:

north:
    hub_connect_url: "http://172.17.0.1:8081"  # double check your machine

You might also need to allow container to hub traffic from your firewall (for example ufw in Ubuntu)

[ka-sarthak]

@DanielYang59 adding the hub_connect_url also worked for me. In addition, I also had to add the nomad-north-jupyter as a dependency of dev-distro. With the new migration of NORTH containers to separate plugins, NORTH containers will not show up in the GUI by default.

[project]
dependencies = [
  ...
  "nomad-north-jupyter",
]