Handle reason='attempt-reconnect' on Linux, and stub for it on macOS/*BSD

The case where the "real" network device disappears or disconnects, while
the VPN/tunnel device stays up is a surprisingly complex one to handle
correctly.  The main issue is that the "explicit route" to the VPN gateway
on the underlying network device may have disappeared, and the OS routing
utilities may erroneously suggest a looped-back route (over the VPN/tunnel
device) as the optimal route to the gateway.

This issue arises particularly — but NOT exclusively — if the *default*
route for the relevant network family is now assigned to the VPN/tunnel
device.

See:

- https://gitlab.com/openconnect/openconnect/issues/17 for the initial report of this problem,
- https://gitlab.com/openconnect/openconnect/-/commit/c2755eefb4e00e915c330495b33d3f5db926615b
  for where the vpnc-script call with reason='attempt-reconnect' was added
  to OpenConnect (merged in v8.02)
- https://gitlab.com/openconnect/vpnc-scripts/-/commit/1000e0f6dd7d6bff163169a46359211c1fc3a6d2
  for where an initial placeholder was first added to vpnc-script
- https://gitlab.com/openconnect/vpnc-scripts/-/merge_requests/14 for the
  first actually-working support in vpnc-script (for Linux)m
- and numerous subsequent changes to handle macOS/*BSD, IPv6, and corner cases

We need to handle attempt-reconnect in vpn-slice.  This mostly borrows from
what vpnc-script does.

Still TODO:

- Flesh out the macOS/*BSD implementation.  Instead of 'route -n get', we should use 'netstat -r -n'
  to ensure that we don't get a looped-back route, as vpnc-script does since
  https://gitlab.com/openconnect/vpnc-scripts/-/blob/412a1faffa72fcda54e8c42d22e0057e56240ff1/vpnc-script#L395-402
- Linux: preserve the 'onlink' route flag.  This requires replacing 'ip route get' with 'ip route show'
  in all cases. See https://gitlab.com/openconnect/vpnc-scripts/-/merge_requests/27

Here is an example of OpenConnect + vpn-slice correctly re-establishing the route
to the VPN gateway even after it's removed by a network outage, leaving the default route looped-back
through the VPN/tunnel interface.

    $ openconnect --script 'vpn-slice --dump -vvv 0.0.0.0/0'
    ...
    ... <authenticate successfully>
    ...
    Called by /usr/sbin/openconnect (PID 123456) with environment variables for vpnc-script:
      reason                  => reason=<reasons.connect: 2>
      VPNGATEWAY              => gateway=IPv4Address('1.2.3.4')
      TUNDEV                  => tundev='tun0'
      ....
    Complete set of subnets to include in VPN routes:
      0.0.0.0/0
    Set explicit route to VPN gateway 1.2.3.4 (via 10.224.0.1, dev wlan0, src 10.224.0.123)
    Blocked incoming traffic from VPN interface with iptables.
    Adding route to nameserver 8.8.8.8 through tun0.
    Adding route to nameserver 1.1.1.1 through tun0.
    Adding route to subnet 0.0.0.0/0 through tun0.                                            # <----
    Added routes for 2 nameservers, 1 subnets, 0 aliases.
    ...
    ... <we have successfully connected>
    ...
    ... <disconnect and reconnect from WiFi, so that explicit route to gateway is lost>
    ...
    ... <OpenConnect dead peer detection kicks in>
    ...
    Failed to reconnect to host vpn.company.com: Interrupted system call
    sleep 10s, remaining timeout 300s
    ...
    Called by /usr/sbin/openconnect (PID 551671) with environment variables for vpnc-script:
      reason                  => reason=<reasons.attempt_reconnect: 5>
      VPNGATEWAY              => gateway=IPv4Address('1.2.3.4')
      TUNDEV                  => tundev='tun0'
      ...
    Complete set of subnets to include in VPN routes:
      0.0.0.0/0
    Reset explicit route to VPN gateway 1.2.3.4 (via 10.224.0.1, dev wlan0, metric 600)       # <---
    SSL negotiation with vpn.company.com
    Connected to HTTPS on vpn.company.com with ciphersuite (TLS1.2)-(ECDHE-SECP256R1)-(RSA-SHA512)-(AES-256-GCM)

Author: dlenski

vpn_slice/__main__.py

@@ -154,6 +154,32 @@ def do_disconnect(env, args):
             print("WARNING: failed to deconfigure domains vpn dns", file=stderr)
 
 
+def do_attempt_reconnect(env, args):
+    # set explicit route to gateway, avoiding loopback through tunnel
+    if not env.gateway:
+        print("WARNING: No gateway address.", file=stderr)
+    elif env.gateway.is_loopback:
+        print("WARNING: Gateway address is loopback (%s); probably a local proxy.", file=stderr)
+    else:
+        # avoid loopback routes through tunnel
+        gwrs = [gwr for gwr in providers.route.get_all_routes(env.gateway) if gwr.get('dev') != env.tundev]
+        if not gwrs:
+            print("WARNING: no routes to VPN gateway found %s; cannot set explicit route to it." % env.gateway)
+            return
+
+        for gwr in gwrs:
+            try:
+                # We do not want to use 'replace', since a route to the gateway that already
+                # exists is mostly likely the correct one (e.g. the case of a reconnect attempt
+                # after dead-peer detection, but no change in the underlying network devices).
+                providers.route.add_route(env.gateway, **gwr)
+            except CalledProcessError:
+                pass
+            else:
+                if args.verbose > 1:
+                    print("Reset explicit route to VPN gateway %s (%s)" % (env.gateway, ', '.join('%s %s' % kv for kv in gwr.items())), file=stderr)
+
+
 def do_connect(env, args):
     global providers
     if args.banner and env.banner:
@@ -594,8 +620,10 @@ def main(args=None, environ=os.environ):
         do_pre_init(env, args)
     elif env.reason == reasons.disconnect:
         do_disconnect(env, args)
-    elif env.reason in (reasons.reconnect, reasons.attempt_reconnect):
-        # FIXME: is there anything that reconnect or attempt_reconnect /should/ do
+    elif env.reason == reasons.attempt_reconnect:
+        do_attempt_reconnect(env, args)
+    elif env.reason == reasons.reconnect:
+        # FIXME: is there anything that reconnect /should/ do
         # on a modern system (Linux) which automatically removes routes to
         # a tunnel adapter that has been removed? I am not clear on whether
         # any other behavior is potentially useful.

vpn_slice/linux.py

@@ -4,7 +4,7 @@
 
 from .posix import PosixProcessProvider
 from .provider import FirewallProvider, RouteProvider, TunnelPrepProvider
-from .util import get_executable
+from .util import MAX_UINT32, get_executable
 
 
 class ProcfsProvider(PosixProcessProvider):
@@ -33,27 +33,50 @@ def _iproute(self, *args, **kwargs):
         for k, v in kwargs.items():
             if v is not None:
                 cl.extend((k, str(v)))
+        skip = next((ii for ii, v in enumerate(args) if not v.startswith('-')), len(args))
+        args = args[skip:]
 
+        output_start = multi = route_junk = None
         if args[:2] == ('route', 'get'):
-            output_start, keys = 1, ('via', 'dev', 'src', 'mtu')
+            output_start, keys = 1, ('via', 'dev', 'src', 'mtu', 'metric')
+            route_junk = True
+        elif args[:2] == ('route', 'show'):
+            output_start, keys = 1, ('via', 'dev', 'src', 'mtu', 'metric')
+            multi = route_junk = True
         elif args[:2] == ('link', 'show'):
             output_start, keys = 3, ('state', 'mtu')
-        else:
-            output_start = None
+
+        # FIXME/rant: parsing the output of 'ip route {get,show}' is a bit of a nightmare.
+        # Per https://linux.die.net/man/8/ip, the output ...
+        #   1) Can include 'mtu' followed by either 1 parameter ('mtu X') or two ('mtu lock X')
+        #   2) Can include 'onlink' followed by 0 parameters, but only in 'ip show' output
+        #      (https://gitlab.com/openconnect/vpnc-scripts/-/issues/20#note_542783676)
+        #
+        # We get both of the above cases wrong currently.
+        # Possibly it'd be saner/stabler to parse /proc/net/route.
 
         if output_start is not None:
-            words = subprocess.check_output(cl, universal_newlines=True).split()
-            if args[:2] == ('route', 'get') and words[0] in ('broadcast', 'multicast', 'local', 'unreachable'):
-                output_start += 1
-            return {words[i]: words[i + 1] for i in range(output_start, len(words), 2) if words[i] in keys}
+            if not multi:
+                words = subprocess.check_output(cl, universal_newlines=True).split()
+                if route_junk and words[0] in ('broadcast', 'multicast', 'local', 'unreachable'):
+                    output_start += 1
+                results = {words[i]: words[i + 1] for i in range(output_start, len(words), 2) if words[i] in keys}
+            else:
+                results = []
+                for line in subprocess.check_output(cl, universal_newlines=True).splitlines():
+                    words = line.split()
+                    if route_junk and words[0] in ('broadcast', 'multicast', 'local', 'unreachable'):
+                        output_start += 1
+                    results.append({words[i]: words[i + 1] for i in range(output_start, len(words), 2) if words[i] in keys})
+            return results
         else:
             subprocess.check_call(cl)
 
-    def add_route(self, destination, *, via=None, dev=None, src=None, mtu=None):
-        self._iproute('route', 'add', destination, via=via, dev=dev, src=src, mtu=mtu)
+    def add_route(self, destination, *, via=None, dev=None, src=None, mtu=None, metric=None):
+        self._iproute('route', 'add', destination, via=via, dev=dev, src=src, mtu=mtu, metric=metric)
 
-    def replace_route(self, destination, *, via=None, dev=None, src=None, mtu=None):
-        self._iproute('route', 'replace', destination, via=via, dev=dev, src=src, mtu=mtu)
+    def replace_route(self, destination, *, via=None, dev=None, src=None, mtu=None, metric=None):
+        self._iproute('route', 'replace', destination, via=via, dev=dev, src=src, mtu=mtu, metric=metric)
 
     def remove_route(self, destination):
         self._iproute('route', 'del', destination)
@@ -66,7 +89,17 @@ def get_route(self, destination):
         if 'dev' in r or 'via' in r:
             return r
 
+    def get_all_routes(self, destination):
+        if destination.version == 4:
+            flag, root = '-4', '0/0'
+        else:
+            flag, root = '-6', '::/0'
+        # Ignore localhost or incomplete routes
+        rs = [r for r in self._iproute(flag, 'route', 'show', 'to', destination, 'root', root) if r.get('dev') != 'lo' and ('dev' in r or 'via' in r)]
+        return sorted(rs, key=lambda r: int(r.get('metric', MAX_UINT32)))
+
     def flush_cache(self):
+        # IPv4 route cache is obsolete as of Linux 3.6 (https://gitlab.com/openconnect/vpnc-scripts/-/merge_requests/30)
         self._iproute('route', 'flush', 'cache')
         self._iproute('-6', 'route', 'flush', 'cache')
 

vpn_slice/mac.py

@@ -43,7 +43,7 @@ def _ifconfig(self, *args):
     def _family_option(self, destination):
         return '-inet6' if destination.version == 6 else '-inet'
 
-    def add_route(self, destination, *, via=None, dev=None, src=None, mtu=None):
+    def add_route(self, destination, *, via=None, dev=None, src=None, mtu=None, metric=None):
         args = ['add', self._family_option(destination)]
         if mtu is not None:
             args.extend(('-mtu', str(mtu)))
@@ -78,6 +78,13 @@ def get_route(self, destination):
                 'mtu': info_d.get('mtu', None),
             }
 
+    def get_all_routes(self, destination):
+        # FIXME: Use netstat to ensure that we don't get a loopback route via the VPN interface itself,
+        # like vpnc-script now does.
+        # https://gitlab.com/openconnect/vpnc-scripts/-/blob/412a1faffa72fcda54e8c42d22e0057e56240ff1/vpnc-script#L395-402
+        route = self.get_route(destination)
+        return [route] if route else []
+
     def flush_cache(self):
         pass
 

vpn_slice/provider.py

@@ -23,7 +23,7 @@ def kill(self, pid):
 
 class RouteProvider(metaclass=ABCMeta):
     @abstractmethod
-    def add_route(self, destination, *, via=None, dev=None, src=None, mtu=None):
+    def add_route(self, destination, *, via=None, dev=None, src=None, mtu=None, metric=None):
         """Add a route to a destination.
 
         You must specify a device or gateway saying where to route to.
@@ -35,7 +35,7 @@ def add_route(self, destination, *, via=None, dev=None, src=None, mtu=None):
         """
 
     @abstractmethod
-    def replace_route(self, destination, *, via=None, dev=None, src=None, mtu=None):
+    def replace_route(self, destination, *, via=None, dev=None, src=None, mtu=None, metric=None):
         """Add or replace a route to a destination.
 
         You must specify a device or gateway saying where to route to.
@@ -52,7 +52,8 @@ def remove_route(self, destination):
 
     @abstractmethod
     def get_route(self, destination):
-        """Return the gateway to a destination.
+        """Return one route to a destination (what the OS considers the
+        optimal or most specific route).
 
         Return a dict with these keys containing the information,
         or None if it is unavailable:
@@ -61,6 +62,23 @@ def get_route(self, destination):
         * dev
         * src
         * mtu
+        * metric
+
+        """
+
+    @abstractmethod
+    def get_all_routes(self, destination):
+        """Return all routes to a destination, sorted in order of decreasing
+        optimality or specificity.
+
+        Return a list of dict with these keys containing the
+        information:
+
+        * via
+        * dev
+        * src
+        * mtu
+        * metric
 
         """
 

vpn_slice/util.py

@@ -2,6 +2,8 @@
 import os.path
 from shutil import which
 
+MAX_UINT32 = 4294967295
+
 
 def get_executable(path):
     path = which(os.path.basename(path)) or path