🐛 BUG: Connections dead after some silence and then need multiple seconds to reestablish

[doits]

What version of nebula are you using? (nebula -version)

1.9.5

What operating system are you using?

Mac

Describe the Bug

Everything works fine until there is some pause. E.g. I connect to host A fine and transfer some data. Then the connections stays silent. When connecting again after some time it takes about ~10 seconds to reestablish the connection.

Logs say the tunnel was dead and needed to be reestablished. It works but the ~10 seconds it takes to reestablish the connection make the command hang.

I guess nebula first needs to detect the connection is dead (takes some time) before it reestablishes it (which works fast according to the logs).

A simple workaround would be to permanently ping some hosts, e.g. a setting like:

permanent_connections:
  host:
    - nebula_ip_address_1
    - nebula_ip_address_2
  interval: 60 # 1 minute

... which means nebula would try to connect to each host every 60 seconds and send a ping, so it detect if the connection is dead and it can be reestablished to be ready to use when needed.

Logs from affected hosts

After the connection was idle and I try to send some date (in this case a simple ping of the nebula host ping 192.168.xxx)

time="2025-04-03T17:29:03+02:00" level=info msg="Tunnel status" certName=xxx localIndex=2635469592 remoteIndex=1129308084 tunnelCheck="map[method:active state:dead]" vpnIp=192.168.xxx
time="2025-04-03T17:29:04+02:00" level=info msg="Handshake message sent" handshake="map[stage:1 style:ix_psk0]" initiatorIndex=2481657292 localIndex=2481657292 remoteIndex=0 udpAddrs="[[xxx::1]:4242 xxx:4242]" vpnIp=192.168.xxx
time="2025-04-03T17:29:04+02:00" level=info msg="Attempt to relay through hosts" localIndex=2481657292 relays="[192.168.xxx]" remoteIndex=0 vpnIp=192.168.xxx
time="2025-04-03T17:29:04+02:00" level=info msg="send CreateRelayRequest" initiatorRelayIndex=3059332896 relay=192.168.xxx relayFrom=192.168.xxx relayTo=192.168.xxx
time="2025-04-03T17:29:04+02:00" level=info msg=handleCreateRelayResponse initiatorRelayIndex=3059332896 relayFrom=3232256116 relayTo=3232256006 responderRelayIndex=1492890361 vpnIp=192.168.xxx
time="2025-04-03T17:29:04+02:00" level=info msg="Handshake message received" certName=xxx durationNs=65953709 fingerprint=a2648f2db9e7b9ac0d59038f0188d59c19c0014f5e72c72a895af4ed1e654f12 handshake="map[stage:2 style:ix_psk0]" initiatorIndex=2481657292 issuer=f5b1a42ed1c1534e597480688e1d8b2f84057b452a8dd4e172d35b10f5d8bc3a remoteIndex=2481657292 responderIndex=2273901310 sentCachedPackets=1 udpAddr="xxx:4242" vpnIp=192.168.xxx

Here's the nebula host on the other side:

time="2025-04-03T17:29:04+02:00" level=info msg="Handshake message received" certName=yyy fingerprint=c60aa76aa4d46ea6835f3536718614c2f2b37e6b3d863d2983596facf92d0bb0 handshake="map[stage:1 style:ix_psk0]" initiatorIndex=2481657292 issuer=f5b1a42ed1c1534e597480688e1d8b2f84057b452a8dd4e172d35b10f5d8bc3a remoteIndex=0 responderIndex=0 udpAddr="yyy:54411" vpnIp=192.168.yyy
time="2025-04-03T17:29:04+02:00" level=info msg="Taking new handshake" certName=yyy localIndex=1129308084 remoteIndex=2635469592 vpnIp=192.168.yyy
time="2025-04-03T17:29:04+02:00" level=info msg="Handshake message sent" certName=yyy fingerprint=c60aa76aa4d46ea6835f3536718614c2f2b37e6b3d863d2983596facf92d0bb0 handshake="map[stage:2 style:ix_psk0]" initiatorIndex=2481657292 issuer=f5b1a42ed1c1534e597480688e1d8b2f84057b452a8dd4e172d35b10f5d8bc3a remoteIndex=0 responderIndex=2273901310 udpAddr="yyy:54411" vpnIp=192.168.yyy
time="2025-04-03T17:29:04+02:00" level=info msg=handleCreateRelayRequest initiatorRelayIndex=2374822517 relayFrom=192.168.yyy relayTo=192.168.yyy vpnIp=192.168.yyy
time="2025-04-03T17:29:04+02:00" level=info msg="send CreateRelayResponse" initiatorRelayIndex=2374822517 relayFrom=192.168.yyy relayTo=192.168.yyy responderRelayIndex=1563939979 vpnIp=192.168.yyy
time="2025-04-03T17:29:15+02:00" level=info msg="Tunnel status" certName=yyy localIndex=1129308084 remoteIndex=2635469592 tunnelCheck="map[method:active state:dead]" vpnIp=192.168.yyy

Config files from affected hosts

pki:
  ca: /Library/Nebula/ca.crt
  cert: /Library/Nebula/host.crt
  key: /Library/Nebula/host.key
  # blocklist is a list of certificate fingerprints that we will refuse to talk to
  #blocklist:
  #  - c99d4e650533b92061b09918e838a5a0a6aaee21eed1d12fd937682865936c72
  # disconnect_invalid is a toggle to force a client to be disconnected if the certificate is expired or invalid.
  disconnect_invalid: true

static_host_map:
  "192.168.xxx": ["xxx:4242", "[xxx]:4242"]

static_map:
  #cadence: 30s

  network: ip

  #lookup_timeout: 250ms

lighthouse:
  am_lighthouse: false
  #serve_dns: false
  interval: 30
  hosts:
    - "192.168.xxx"

listen:
  # To listen on both any ipv4 and ipv6 use "::"
  host: "::"
  port: 0
  #send_recv_error: always

punchy:
  punch: true
  respond: true

  # delays a punch response for misbehaving NATs, default is 1 second.
  #delay: 1s

  # set the delay before attempting punchy.respond. Default is 5 seconds. respond must be true to take effect.
  #respond_delay: 5s

#cipher: aes

relay:
  relays:
    - 192.168.xxx
  am_relay: false
  use_relays: true

# Configure the private interface. Note: addr is baked into the nebula certificate
tun:
  # When tun is disabled, a lighthouse can be started without a local tun interface (and therefore without root)
  disabled: false
  #dev: nebula1
  drop_local_broadcast: false
  drop_multicast: false
  tx_queue: 500
  mtu: 1300

  #use_system_route_table: false

# TODO
# Configure logging level
logging:
  # panic, fatal, error, warning, info, or debug. Default is info
  level: info
  # json or text formats currently available. Default is text
  format: text


# Handshake Manager Settings
#handshakes:
# Handshakes are sent to all known addresses at each interval with a linear backoff,
# Wait try_interval after the 1st attempt, 2 * try_interval after the 2nd, etc, until the handshake is older than timeout
# A 100ms interval with the default 10 retries will give a handshake 5.5 seconds to resolve before timing out
#try_interval: 100ms
#retries: 20
# trigger_buffer is the size of the buffer channel for quickly sending handshakes
# after receiving the response for lighthouse queries
#trigger_buffer: 64

# Nebula security group configuration
firewall:
  # Action to take when a packet is not allowed by the firewall rules.
  # Can be one of:
  #   `drop` (default): silently drop the packet.
  #   `reject`: send a reject reply.
  #     - For TCP, this will be a RST "Connection Reset" packet.
  #     - For other protocols, this will be an ICMP port unreachable packet.
  outbound_action: reject
  inbound_action: reject

  conntrack:
    tcp_timeout: 12m
    udp_timeout: 3m
    default_timeout: 10m

  outbound:
    # Allow all outbound traffic from this node
    - port: any
      proto: any
      host: any

  inbound:
    # Allow traffic between any nebula hosts
    - port: any
      proto: any
      host: any

[d4l3k]

I also have been experiencing this and it's super annoying when it happens.

I run Nebula on both my car (Tesla Model 3) and my phone as well as for things like home assistant. Often when I need nebula I'm on the move and the network switches between Wi-Fi and LTE. Usually I'll manually restart Nebula on my phone which works but this happens multiple times per week

Couple of examples:

• Leaving my house, I pull out of my garage, I lose Wi-Fi and then have to wait for Nebula to reconnect before I can close my garage door
• Leaving work I'll pull out my phone on the way to the car to turn on the heat and I'll have to wait for Nebula

[dmurray14]

I agree, this is extremely annoying and a fix would be awesome. It's one of the things that is currently stopping us from migrating over some web apps - many users roam and won't understand why the site "never loads the first time" (while Nebula is waiting to timeout). Generally, I know when it's going to happen and it's quicker to just restart dnclient than wait for the timeout, but that's not really a solution for end users.

Having this either configurable (something like 3-5 seconds probably would be tolerable) or avoiding it altogether somehow would be a big win.

[johnmaguire]

@Cyberes This issue is related to the fact that Nebula doesn't require keepalive packets to consider the tunnel alive. If traffic isn't flowing, but the tunnel has broken, Nebula currently has no way of finding out.

#1413 likely closes this ticket, but it has currently only been merged to release-1.9.

[johnmaguire]

Closed with #1427.