Help me to understand the HTTPS issues

[wsw70]

The Start

As my quest to have TSDProxy working continues, I am now faced with HTTPS issues I would like to clarify (not What is HTTPS? but Why is HTTPS involved here?)

I started with one service (Jellyseerr) which I decorated with labels:

    labels:
      - tsdproxy.enable=true
      - tsdproxy.name=jellyseerr
      - tsdproxy.container_port=5055

TSDProxy started, ultimately with an error message:

tdsproxy-tsdproxy-1  | 6:21PM ERR Error Listening on TLS error="tsnet: you must enable HTTPS in the admin panel to proceed. See https://tailscale.com/s/https" module=proxymanager proxyname=media-jellyseerr-1

My first question was: why? What is trying to connect with what so that HTTPS is required?

Enabling HTTPS

I went to the admin panel as instructed, read about HTTPS support and from what I understand, Tailscale gives the ability to use LE certificates so that connecting to a HTTP endpoint on the Tailscale machine does not raise certificate warnings. Fair enough, this is a nice addition to the ones who would like that. It however requires each machine (in my case - each container) to run tailscale cert to initiate the LE certificate generation and to manage the continuous reissuing every 90 days.

OK, but what does this have to do with TSDProxy? At what point TDSProxy even need to connect to the service with HTTPS? (when the service is HTTP). If I get it right, its only role (and a great, awesome one!) is to register the service in the tailnet and proxy (see last paragraph, though) - no need for HTTPS on the service for that.

I enabled certificates in the admin panel and the connection to the services (via the tailnet FQDN) fails with a timeout. Note that I have not yet issued tailscale cert so there is no LE certificate for this service, but I could expect a missing or incorrect cert, but not a timeout

Further considerations

jellyseerr is registered in tailnet with 100.108.29.54. srv, my main server where docker runs is at 100.79.183.102. So when type jellyseerr.funky-name.ts.net it is resolved to a CGNAT IP which actually points to ... I do not know to what. My Jellyseerr container has an internal docker IP and that's all.

Sure, a proxy could forward the HTTP call to that IP but for that, it would need to be resolved to jellyseerr.funky-name.ts.net (as this is the case with a normal edge router). But TSDProxy is a container on srv...

---

Unrelated note: I work in IT for 30 years (in cybersecurity and disruptive technologies) and I am always curious about new tech. TSDProxy is a tool that really, really triggered my curiosity because for the first time in quite a long time I cannot wrap my head around how it works :) Thanks @almeidapaulopt for that! 👍

[stevetrease]

TSDProxy creates a Tailscale "machine" and requests a LE certificate for each proxied container. Each "machine" will have unique Tailscale hostname and Tailscale CGNAT IP address which are terminated by TSDProxy which forwards the required traffic to the appropriate docker container over the local docker internal network.
It's kind of magic :-)
Have a look at the diagram in the documentation.

Note that you don't need to request any tailscale certs - if TSDProxy is working properly it will request them on your behalf.

What is trying to connect with what so that HTTPS is required?

TSDProxy is resting to request the certificates from Lets Encrypte via Tailscale. It can't get the certificate if https isn't enabled for your tailnet in the Tailscale console.

[stevetrease]

Each proxied docker container has a Tailscale "machine" with a unique Tailscale "hostname" and IP address. The TSDProxy container is responding to all those IP addresses.

What is https needed for? Securing the traffic from the client, over Tailscale to the TSDProxy process.

So, imagine a PC on a LAN )or somewhere on the the internet), let's call it client1; a host running tsdproxy in a container (let's call it tsdproxy1) and two web servers (web1 and web 2) running in other docker containers on the same host.

client1 is a member of the tailnet.
tsdproxy1 is a member of the tailnet.
When the web1 and web2 docker containers start, tsdproxy1 detects them, and creates two new tailscale "machines", creates a LE cert for each and then starts to listen for HTTPS traffic for web1.funnyname.ts.net and web2.funnyname.ts.net.

So, client1 trues to access a webpage on https://web1.funntname.ts.net.
Client1 uses Tailscale's magicDNS to resolve the tailscale IP address for https:/web1.funnyname.ts.net.
The request is sent via the tailnet to that IP address where TSDProxy is listening on port 443.
TSDProxy decrypts the request and forwards it over the docker network to the web1 docker container.
web1 process the response and sends the response back to the TSDProxy process which encryptes the reply and sends it over the tailnet back to client1 where the webpage is shown.

So https is used between the client and tsdproxy.

Does that make sense?

[stevetrease]

Maybe this log extract from my tsdproxy container will help.

I run a container called esphome that exposes a web UI on port 6052.

docker compose logs -ft | grep esphome

If you lookyou can see TSDProxy detecting the container and it's docker network address (172.17.0.1) in this case.
Then starting tsnet with a hostname of esphome.
The final line is one of my clients connecting to esphome.funnyname.ts.net and passing the request to the esphome docker container.

tailscale-docker-proxy-1  | 3:28PM INF Trying to auto detect target URL container=/esphome docker=local module=proxymanager try=0
tailscale-docker-proxy-1  | 3:28PM INF Successfully connected using defaultBridgeAddress and internal port address=172.17.0.1 container=/esphome docker=local module=proxymanager port=6052
tailscale-docker-proxy-1  | 3:28PM INF setting up proxy hostname=esphome module=proxymanager proxyname=esphome
tailscale-docker-proxy-1  | 3:28PM INF starting proxy module=proxymanager name=esphome proxyname=esphome
tailscale-docker-proxy-1  | 3:28PM INF tsnet running state path /data/ts-default/esphome/tailscaled.state Hostname=esphome module=proxymanager tailscale=ts-default
tailscale-docker-proxy-1  | 3:28PM INF tsnet starting with hostname "esphome", varRoot "/data/ts-default/esphome" Hostname=esphome module=proxymanager tailscale=ts-default
tailscale-docker-proxy-1  | 3:28PM INF LocalBackend state is NeedsLogin; running StartLoginInteractive... Hostname=esphome module=proxymanager tailscale=ts-default
tailscale-docker-proxy-1  | 3:29PM INF AuthLoop: state is Running; done Hostname=esphome module=proxymanager tailscale=ts-default
tailscale-docker-proxy-1  | 3:30PM INF request host=esphome.tail5e5d.ts.net method=GET module=proxymanager proxyname=esphome status=304 url=/

[wsw70]

The TSDProxy container is responding to all those IP addresses

Ahhh, this explains that TSDProxy handles the termination and proxification. I do not know how it responds to these IPs but let's say that this is "Tailscale Magic (C)" (I worked on the WG protocol so I will dig for the magic part in the sources :))

OK, so now it is clear that TSDProxy responds to the calls for the services - let's go to the HTTPS part.

What is https needed for? Securing the traffic from the client, over Tailscale to the TSDProxy process.

This is just a nice to have, on several levels.

• First you may not want to create a HTTPS tunnel just because. It would be equivalent to connect to the ESPHome dashboard via http://the_docker_host:6052 from the network where the_docker_host is (big fan of ESPHome as well).
• Then you may want to use the Wireguard tunneling (which is the basis for Tailscale) to secure your call and then access http://esphome.funny-name.ts.net (again HTTP not HTTPS)
• And finally you may want to have the proxy (TSDProxy) expose the service on 443 (itself), manage the certs from LE etc. - some of the things an edge router would do (traefik, caddy, ...)

How can I have case number two? (it's not that I do not like the full-fledged HTTPS, far from that, I want to understand why HTTPS would be a requirement)

[stevetrease]

You want to proxy your container with TSDProxy but not but https?

I don't think you can at the moment, TSSDProxy only exposes https endpoints.

However, it is on the TSDproxy roadmap and shows as complete.
I assume that Paulo is currently testing it and will release it when he is happy with it.

[stevetrease]

Further to understanding how TSDProxy works, under the hood it uses Tailscale's tsnet Go library for it's magic.

It is worth having a read of Tailscales document here:

Typically, every IP address in your tailnet has to be strongly associated with a single machine. This makes it difficult to run multiple services on a single machine. Especially if all of those services have different access control rules. tsnet lets you run multiple services on the same machine with different IP addresses, access control rules, and even user identities.

[almeidapaulopt]

some times a image helps better.