How to use with docker swarm?

[hacketaler]

Hi
I'm using docker swarm with 9 nodes and have been struggling with networking, tailscale sidecars/macvlan, https several times. So this solutions sounds great.
But do I need a tsdproxy on every node, as swarm normally manages redirection between nodes.
Could anybody already test it with swarm?

[UnclePhil]

Not working, because in swarm you need to use service labels, and not container labels.
i know we need this feature , so push my feature request (#63)

[almeidapaulopt]

I have it running with swarm, have you tested?
Could be some configurations different than mine.

[UnclePhil]

This my test stack .

version: "3.7"
services:
  draw:
    ## image: unclephil/draw.io:latest
    image: fjudith/draw.io:alpine
    networks:
      - proxy
    deploy:
      labels:
        - caddy=http://${HOST}.${CLUSTER_DOMAIN}
        - caddy.reverse_proxy={{upstreams 8080}}
        - caddy.encode=gzip
        - caddy.header.etrace=${CLUSTER_NAME},draw
        - caddy.header.-Server
        - caddy.header.-X-Forwarded-For
        - caddy.header.-x-powered-by
        # - caddy.tls=${TLS_EMAIL}
        - caddy.log.output=stdout

       ## homepage
        - homepage.group=${CLUSTER_NAME}
        - homepage.name=Draw
        - homepage.description=Draw.io editor
        - homepage.icon=draw-io.png
        - homepage.href=http://${HOST}.${CLUSTER_DOMAIN}
        - homepage.weight=20

        ## TSDproxy
        - tsdproxy.enable=true
        - tsdproxy.name=draw
        - tsdproxy.container_port=8080
        - tsdproxy.ephemeral=true
        
      replicas: 1
      resources:
        limits:
          memory: 512M
        reservations:
          memory: 64M
      update_config:
        parallelism: 1
        delay: 10s
        order: start-first
      restart_policy:
        condition: on-failure

You can see that in a swarm stack, it's better to put labels at service level, because tasks can move from one host to another without warning.
When i read your code, i see that you are reading container labels who are on the same host than tsdproxy.
In this case, if a task move to another host, the connection is lost.

That's why, for swarm people, we need to read the service labels in place of the container.

No need of combined reading.
Set a global "RunOnSwarm" flag, and then process a service query.

I'm not going to propose a PR because my GO is very basic ( i'm a bash, python and node writer) but the Go docker api have all the necessary features to adapt your code

[almeidapaulopt]

you could add a tsdproxy per server. like this https://almeidapaulopt.github.io/tsdproxy/docs/scenarios/2i-2docker-1tailscale/

when a container moves it will start normally on the other container.

[UnclePhil]

Thats right, but how did you proceed when you have 8 replicas in the cluster and 2 or 3 on the same host. ..... with the same set of labels ??

Thats the real reason of service labels.

[hacketaler]

sorry, spend some time with other services...
Despite a very similar setup working perfectly on a standalone Docker host, my Swarm-based configuration doesn't function as expected. The service registers in Tailscale and appears in the Tailscale admin console, but requests to the Tailscale domain fail or behave inconsistently.

My working setup on a single host:

services:
  tsdproxy_lenny:
    image: almeidapaulopt/tsdproxy:latest
    ports:
      - "8041:8080"
    volumes:
      - /volumesOMV/tsd-proxy_tailscale-rp/len64gb02:/config
      - tsdproxy-tailscale:/data
      - /var/run/docker.sock:/var/run/docker.sock
    restart: unless-stopped

volumes:
  tsdproxy-tailscale:
    driver: local

defaultproxyprovider:` default
docker:
    local:
        host: unix:///var/run/docker.sock
        targethostname: 172.31.0.1
        defaultproxyprovider: default
tailscale:
    providers:
        default:
            authkey: tskey-auth-***
            controlurl: https://controlplane.tailscale.com
    datadir: /data/
http:
    hostname: 0.0.0.0
    port: 8080
log:
    level: info
    json: false
proxyaccesslog: true

startet nginx with this:

docker run -d --name sample-lenny \
  -p 8113:80 \
  --label "tsdproxy.enable=true" \
  --label "tsdproxy.name=lastone" \
  --label "tsdproxy.container_port=80" \
  nginx:latest

===>
The Tailscale URL https://lastone.***.net/ works perfectly.
Logs show successful proxy setup
INF Successfully connected using docker network gateway and exposed port address=172.17.0.1

NOT WORKING WITH SWARM:

services:
  tsdproxy_swarm:
    image: almeidapaulopt/tsdproxy:latest
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - tsdproxy-tailscale:/data
      - /volumesOMV/tsd-proxy_tailscale-rp:/config
    deploy:
      mode: global
      restart_policy:
        condition: any

volumes:
  tsdproxy-tailscale:
    external: true

defaultproxyprovider: default
docker:
    local:
        host: unix:///var/run/docker.sock
        targethostname: 172.31.0.1
        defaultproxyprovider: default
tailscale:
    providers:
        default:
            authkeyfile: "/config/key_tsdproxy.key"
            controlurl: https://controlplane.tailscale.com
    datadir: /data/
http:
    hostname: 0.0.0.0
    port: 8080
log:
    level: info
    json: false
proxyaccesslog: true

services:
  nginx-test:
    image: nginx:latest
    ports:
      - "8115:80"
    labels:
      tsdproxy.enable: "true"
      tsdproxy.name: "nginx-swarm3"
      tsdproxy.container_port: "80"

===>
The service appears in Tailscale.
However, the Tailscale domain https://nginx-swarm3.***.net/ does not work.
Logs show the container is detected and registered, but there are TLS handshake errors:
TLS handshake error from 100.119.***.13: no SNI ServerName

tried also with this:
tsdproxy.scheme: "https"
tsdproxy.tlsvalidate: "false"
==> no success

How can I debug further to identify why requests to the Tailscale domain fail in the Swarm setup?
Thank you for your help! Any guidance would be greatly appreciated.

[UnclePhil]

To be connectable, both service must be on the same network. This is the base rules in swarm.
and if the network is not declared swarm create a "default_" for each service.
So in your swarm case,

• create a tsdproxy network with attachable=true
• edit both stackfile to add the networks section with the tsdproxy
• reload both stack

docker --context $CLUSTER_NAME network create tsdproxy -d overlay --attachable

services:
  tsdproxy_swarm:
    image: almeidapaulopt/tsdproxy:latest
    networks:
      - tsdproxy
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - tsdproxy-tailscale:/data
      - /volumesOMV/tsd-proxy_tailscale-rp:/config
    deploy:
      ## load only one instance
      # mode: global
      restart_policy:
        condition: any
     ## on manager 
      placement:
        constraints:
          - node.role == manager
networks
  tsdproxy:
     external: true
volumes:
  tsdproxy-tailscale:
    external: true

services:
  nginx-test:
    image: nginx:latest
    networks:
      - tsdproxy
    ports:
      - "8115:80"
    labels:
      tsdproxy.enable: "true"
      tsdproxy.name: "nginx-swarm3"
      tsdproxy.container_port: "80"
networks
  tsdproxy:
     external: true

BTW:

• in this case the mode:global from the tsdproxy stack can be removed because your network is present on all nodes but the instance must be fixed on a manager
• It's better to load the config file via a docker config

The only problem on this configuration is the case you have multiple replica of the nginx services because , due to the container labels, you are declaring in TS , N times the same host name . And this will be tricky .

That's why i ask the ability to use also services labels.

Have fun,
Swarm Rocks !

[hacketaler]

thank you. network was key. but I still deploy global ;)