Docker swarm integration

[Kegelcizer]

After setting up the docker integration, in the host configuration Host details -> Routing -> Option / App, I can only see containers or swarm services with published ports as host.

[lucasdillmann]

Hi @Kegelcizer, hope you're doing well.

I've revised the code for the Docker integration and indeed, it unnecessarily limits options to only ports exposed on the host. I'm working on an improvement to show ports exposed in the container as options too.

The new version should be available soon.

[lucasdillmann]

@Kegelcizer, both the Docker integration update (allowing selection of container ports) and the clickable domain links on the hosts list page are now available in version 2.24.0. Thanks again for the suggestions, and please let me know if anything doesn't work as expected.

[lucasdillmann]

In swarm, any service can be accessed from any node directly by IP:PORT (this means that port is used on all nodes), and the mesh will route to the actual container, you do not need to "grab" any data (ip or port) from the actual container.

This is actually why I don't think the current integration with Docker will work as expected. Ignition is currently only capable of grabbing the IP and port from the container and not from the load balancer. It doesn't even know that multiple nodes could exist or that it needs to reach the service through the balancer in those scenarios.

explicitly set a node on the 3rd integration as target, when configuring the integration OR for even more flexibility set this at host details -> routing entry level for this integration type.

That's the path I think Ignition needs to follow as well, but by providing the list of services deployed in the cluster within the UI.

By reading the services deployed in the Swarm instead of listing individual containers, I can in theory get the endpoint to reach them through the balancer. Such endpoints seem to be the port and a DNS name that will resolve to all the node IPs. Since nginx is capable of resolving DNS at runtime (using Swarm as the DNS server), this allows the connection to be made without needing to explicitly use the node IP+port in the Nginx config files. This means a node could be completely replaced (and its IP changed) without needing to update or reload anything in Nginx or Nginx Ignition.

By doing this, Ignition will be aware of the Swarm Service Mesh and use it to reach services, taking full advantage of the features the container orchestration provides (including Nginx balancing requests across Swarm's own balancers).

---

Much of my understanding so far is based on Docker documentation and my experience with Kubernetes' Service Mesh, which is why I plan to test and study this further to ensure it works well with Swarm. I should have more details and even a beta version to share quite soon.

[lucasdillmann]

Hi @Kegelcizer, hope you're doing well.

I'm still validating some minor details before launching the final version, but the dillmann/nginx-ignition:pr-60-snapshot beta version is now available if you'd like a preview of the native support for Docker Swarm.

In the integrations form, Ignition now features a new checkbox for Docker Swarm. When enabled, the host form will display a list of services with ingress and/or host mappings instead of individual containers.

Ignition will resolve and use the leader host's IP address by default, but if the app is running inside your Swarm cluster, there is also an option to enable Service Mesh. This makes Ignition configure Nginx to reach services using Docker's internal DNS names (when an ingress was selected), effectively decoupling the traffic from the leader node IP.

Please let me know if you find any unexpected behaviors. I'll update the thread here when the final version is out.

[lucasdillmann]

@Kegelcizer, version 2.25.0 is now out with the changes above and other minor improvements.

I hope Ignition now better integrates with your Swarm cluster. Please let me know if not, and thanks again for the suggestion.

[Kegelcizer]

Hi @lucasdillmann, today I had some time to test it and I have some issues:

1. Using service mesh, selecting a service from other node that is not ignition:

2. Not using service mesh, selecting a service from other node that is not ignition yields 502 Bad Gateway

Here's my test swarm:

Please let me know if more details are needed.

[lucasdillmann]

@Kegelcizer, I've reviewed the information you shared, ran some additional tests, and found that:

1. I made an incorrect assumption that Ignition could always use the leader node IP when a host binding was selected. Only a node with at least one running task can be used.
2. In Service Mesh mode, Ignition was generating target URLs like http://stack-name_service-name:8080, where 8080 was the published port. The URL should actually be generated using the target/container port (I didn't catch this in my initial tests because I was using the same value on both).

Version 2.25.1 is now out with fixes for both of these issues. Thanks for sharing these details with me.

Regarding the "host not found" error in your first image, this is likely related to the network configuration of the containers. Without a shared overlay network between Ignition and the other services, Nginx will fail to resolve the DNS because Docker won't expose those service names to the Ignition container.

For reference, I've attached the test compose files I used where the Service Mesh worked as expected. My test cluster consisted of three VMs, with two of them acting as worker nodes.

hello-swarm.docker-compose.yaml
nginx-ignition.docker-compose.yaml

[Kegelcizer]

Tested again on the same network, service mesh toggle works as advertised. Added 1 more worker, moved container on that one without changing ignition host settings, works!
Once again, thank you for the implementation, fast fixes and helping.
Happy holidays!

[lucasdillmann]

Thanks for the feedback, @Kegelcizer. Happy to hear that it's working as expected now. Happy holidays!

[Kegelcizer]

Hi @lucasdillmann, everything worked fine till I got to locations :)
I'm not sure if I did something wrong or there's a bug or two.

Here I can see it saved the containerId:port instead of serviceName:port
Maybe it's irrelevant since the generated configuration looks ok (more on point 2)

Down here we can see the correct serviceName:port under /externalapi/ location, however it seems there is a missing trailing / on proxy_pass.
I could not make any location work without leading and trailing / on location declaration and trailing / on proxy_pass on any other proxy type, so I guess this might be the case on integration type.

server {
    root /dev/null;
    access_log "/tmp/nginx-ignition/nginx/logs/host-cdb55168-8174-47e5-835a-01e37a2cd453.access.log";
    error_log "/tmp/nginx-ignition/nginx/logs/host-cdb55168-8174-47e5-835a-01e37a2cd453.error.log" warn;
    gzip on;
    client_max_body_size 1024M;
    
    
    
    http2 on;
    
    listen 0.0.0.0:443 ssl ;
    ssl_certificate "/tmp/nginx-ignition/nginx/config/certificate-479319ad-8294-4bb0-b418-e75f858b29c9.pem";
    ssl_certificate_key "/tmp/nginx-ignition/nginx/config/certificate-479319ad-8294-4bb0-b418-e75f858b29c9.pem";
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers HIGH:!aNULL:!MD5;
    
    server_name external.dev.myDomain.com;
    location / {
        proxy_pass http://npm1.staging.myDomain.com:80;
        proxy_set_header Host npm1.staging.myDomain.com:80;
        
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        
        proxy_ssl_server_name on;
        proxy_set_header x-forwarded-for $proxy_add_x_forwarded_for;
        proxy_set_header x-forwarded-host $host;
        proxy_set_header x-forwarded-proto $scheme;
        proxy_set_header x-forwarded-scheme $scheme;
        proxy_set_header x-forwarded-port $server_port;
        proxy_set_header x-real-ip $remote_addr;
        
    }
    location /externalapi_old/ {
        proxy_pass http://services1.staging.myDomain.com:5049/;
        proxy_set_header Host services1.staging.myDomain.com:5049;
        
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        
        proxy_ssl_server_name on;
        proxy_set_header x-forwarded-for $proxy_add_x_forwarded_for;
        proxy_set_header x-forwarded-host $host;
        proxy_set_header x-forwarded-proto $scheme;
        proxy_set_header x-forwarded-scheme $scheme;
        proxy_set_header x-forwarded-port $server_port;
        proxy_set_header x-real-ip $remote_addr;
        
    }
    location /externalapi/ {
        resolver 127.0.0.11 valid=5s;
        proxy_pass http://dev-myStack_externalapi:8080;
        proxy_set_header Host dev-myStack_externalapi:8080;
        
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        
        proxy_ssl_server_name on;
        proxy_set_header x-forwarded-for $proxy_add_x_forwarded_for;
        proxy_set_header x-forwarded-host $host;
        proxy_set_header x-forwarded-proto $scheme;
        proxy_set_header x-forwarded-scheme $scheme;
        proxy_set_header x-forwarded-port $server_port;
        proxy_set_header x-real-ip $remote_addr;
        
    }
}

Thank you!

[lucasdillmann]

Hi @Kegelcizer. Regarding the first item, Nginx Ignition actually saves the Service ID when Swarm mode is active. This enables you to change the Swarm service name without "breaking" the Host configuration (as long Docker don't change the service ID). When the Nginx configuration is reloaded, Ignition resolves the name linked to the service via its ID and places it into the config file.

As for the location part, changing the path on the target (which includes placing a simple /) is not yet supported when using an integration (Docker, Swarm, or TrueNAS). I was aware of this, but it hadn't been prioritized yet (I didn't need it personally and it wasn't on my radar that anyone else did).

That being said, I can improve this and it shouldn't take long. Give me a bit of time and I’ll get back to you with an update.

[lucasdillmann]

It's ready, @Kegelcizer. Version 2.30.0 enables you to customize the base URI on the target service on integration routes. A new field should show up in the UI where you can place the missing trailing slash (or any other base path).

Hope this helps. Please let me know if it doesn't.

[Kegelcizer]

Added the /, works now. Thanks again!