Skip to content

feat(discv4): migrate DiscV4 UDP transport from Vert.x to Netty#10716

Open
usmansaleem wants to merge 8 commits into
besu-eth:mainfrom
usmansaleem:netty-discv4-transport
Open

feat(discv4): migrate DiscV4 UDP transport from Vert.x to Netty#10716
usmansaleem wants to merge 8 commits into
besu-eth:mainfrom
usmansaleem:netty-discv4-transport

Conversation

@usmansaleem

@usmansaleem usmansaleem commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

PR description

This PR ports the DiscV4 peer discovery UDP layer from Vert.x to Netty,
replacing VertxPeerDiscoveryAgent and VertxTimerUtil with a clean,
Vert.x-free implementation. It is a focused extraction from #10624 so
that the foundational transport change can be reviewed and merged
independently.

What changes

New files

  • V4Transport — interface abstracting UDP send/receive from the agent
  • transport/NettyV4Transport — Netty NIO UDP channel implementation (IPv4)
  • NettyPeerDiscoveryAgent — concrete agent wiring ScheduledExecutorService
    threads for timers and crypto
  • internal/ScheduledExecutorTimerUtil — replaces VertxTimerUtil
  • internal/ScheduledExecutorAsyncExecutor — replaces the Vert.x worker pool

Deleted files

  • VertxPeerDiscoveryAgent
  • internal/VertxTimerUtil

Modified

  • PeerDiscoveryAgentV4 — ported to V4Transport interface; added
    dispatchExecutor ordering guarantee (timer callbacks and inbound packet
    dispatch share a single thread, preserving the Vert.x event-loop ordering);
    a dedicated single-thread decode executor was added afterward (see "Fixes
    from review")
  • PeerDiscoveryAgentFactoryV4 — wires NettyPeerDiscoveryAgent
  • DefaultPeerDiscoveryAgentFactory — updated to match
  • PeerDiscoveryController — adds a dispatchExecutor used to run outbound
    packet-creation callbacks on the same thread as inbound packet handling
    (see "Fixes from review")
  • ethereum/p2p/build.gradle — adds netty-transport and netty-handler
  • NodeRecordManager — fixes two ENR equality bugs (compressed pubkey
    comparison; fork-id wrapper list shape) that caused seq to increment on
    every restart even when nothing changed
  • PingPacketData.getTo() — returns Optional<Endpoint> for EIP-8
    compliance (malformed to field must not prevent packet processing)
  • Test classes updated throughout for the above API changes

Fixes from review

Since the initial migration, this PR picked up several concurrency/
correctness fixes from Copilot's automated review, a full manual code
review, and a live mainnet dry-run used to validate startup/shutdown
behavior:

  • Outbound/inbound packet handling race closed: PeerDiscoveryController .createPacket()'s post-signing continuation now runs on the same
    single dispatch executor used for inbound packet matching, instead of
    whichever crypto-pool thread finished signing — this was a real race on
    PeerInteractionState's response-matching filter (a fast-arriving PONG
    could be matched before the filter update was visible).
  • Inbound packet decode ordering restored: PeerDiscoveryAgentV4 now
    uses a dedicated single-thread decode executor
    (createDecodeExecutor()), separate from the 2-thread signing pool,
    restoring the packet arrival-order guarantee the removed Vert.x
    implementation provided via an ordered executeBlocking.
  • Shutdown race closed: NettyPeerDiscoveryAgent.stop() now runs
    PeerDiscoveryController.stop() on timerScheduler itself rather than
    whatever Netty thread completes transport.stop(), closing a race with
    in-flight timer/dispatch work during shutdown.
  • PING expiry validation restored: PingPacketDataRlpReader was
    rewritten to bypass PingPacketDataFactory (to tolerate a malformed to
    field per EIP-8), which also silently dropped the factory's expiry check.
    Expiry is now validated directly in the reader; malformed-to tolerance
    is unaffected.
  • NettyV4Transport hardening:
    • UDP bind failures are wrapped in PeerDiscoveryServiceException with a
      descriptive host:port message again, matching the old Vert.x behavior.
    • The DEBUG/ERROR severity split in the inbound channel exception handler
      is restored (previously collapsed to unconditional DEBUG).
    • The event-loop group (and NettyPeerDiscoveryAgent's timer/crypto/decode
      executors) are now created lazily in start() instead of eagerly in the
      constructor, so a node running with discovery disabled carries no idle
      discv4 threads.
    • Shutdown uses an explicit short quiet period instead of Netty's 2s
      default — confirmed via a live mainnet run that this was adding ~2s to
      every node shutdown; it now completes in ~10-15ms.
  • Endpoint EIP-8 decoding hardened: maybeDecodeStandalone()'s catch
    was broadened from IllegalPortException to any RuntimeException, so
    any malformed to field is tolerated, not just an out-of-range port.
    decodeStandalone() was also fixed to skip to the end of the malformed
    RLP list before leaving it, keeping the cursor in sync with the enclosing
    list regardless of where decoding failed.
  • Executor edge cases: ScheduledExecutorAsyncExecutor now completes
    its future exceptionally instead of throwing synchronously when the
    underlying executor rejects a task (e.g. during shutdown);
    ScheduledExecutorTimerUtil now catches exceptions from periodic
    handlers so one failure doesn't permanently cancel the scheduled task.
  • Test hardening: MockPeerDiscoveryAgent's mock transport no longer
    risks matching a stopped/replaced agent at a reused address, and its
    constructor-chain/mutable-array back-reference hack was replaced with a
    post-construction setter mirroring the production setInboundHandler
    pattern.

Limitations / known constraints

  • IPv4-only transport: NettyV4Transport is pinned to
    SocketProtocolFamily.INET. Peers that advertise an IPv6-only address
    are silently skipped (TRACE log). Dual-stack IPv4+IPv6 peers work
    normally via their IPv4 address. This is an intentional temporary
    restriction; the follow-on feat(p2p): DiscV4 + DiscV5 on shared UDP socket with --discovery-mode selector #10624 introduces a shared dual-stack channel
    that lifts this limitation.
  • DNS discovery still uses Vert.x: DNSDaemon / DNSResolver remain
    Vert.x-based. The ethereum/p2p module retains its vertx-core
    dependency for that subsystem.
  • vertx_eventloop_pending_tasks metric dropped: That gauge was
    Vert.x-specific and has no direct equivalent. All other discovery metrics
    (besu_network_discovery_*) come from PeerDiscoveryController and are
    unaffected.

Fixed Issue(s)

Thanks for sending a pull request! Have you done the following?

  • Checked out our contribution guidelines?
  • Considered documentation and added the doc-change-required label to this PR if updates are required.
  • Considered the changelog and included an update if required.
  • For database changes (e.g. KeyValueSegmentIdentifier) considered compatibility and performed forwards and backwards compatibility tests

Locally, you can run these tests to catch failures early:

  • spotless: ./gradlew spotlessApply
  • unit tests: ./gradlew build
  • acceptance tests: ./gradlew acceptanceTest
  • integration tests: ./gradlew integrationTest
  • reference tests: ./gradlew ethereum:referenceTests:referenceTests
  • hive tests: Engine or other RPCs modified?

Replaces VertxPeerDiscoveryAgent and VertxTimerUtil with a Netty-backed
implementation using NioDatagramChannel (pure Java, no native epoll).

- NettyPeerDiscoveryAgent: wires ScheduledExecutorService for timers and
  a fixed-2-thread pool for crypto work; dispatches inbound packets on the
  timer thread to preserve single-threaded event-loop ordering
- NettyV4Transport: binds an IPv4 NioDatagramChannel, encodes/decodes UDP
  frames via Netty pipeline, surfaces start/stop/send as CompletableFutures
- ScheduledExecutorTimerUtil / ScheduledExecutorAsyncExecutor: drop-in
  replacements for the Vert.x timer and worker abstractions
- stopGate (AtomicBoolean) in PeerDiscoveryAgentV4 prevents sends and
  swallows errors after stop() is called
- IPv6 peers are silently skipped on the IPv4-only transport (TRACE log)
- Removes Vert.x dependency from ethereum:p2p; adds netty-transport and
  netty-handler
- Fixes ENR equality bugs in NodeRecordManager (Bytes vs hex comparison)
- Adds NettyPeerDiscoveryAgentTest covering stopGate and all
  handleOutgoingPacketError branches; adds NettyV4TransportTest and
  ScheduledExecutorTimerUtilTest

Closes besu-eth#10624

Signed-off-by: Usman Saleem <usman@usmans.info>
@usmansaleem usmansaleem force-pushed the netty-discv4-transport branch from 5b6aa88 to ec79189 Compare June 29, 2026 08:14
Copilot AI review requested due to automatic review settings July 2, 2026 01:21

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR migrates the DiscV4 UDP discovery transport from a Vert.x-based implementation to a Netty-based transport with a new V4Transport abstraction, aiming to preserve Vert.x event-loop ordering semantics while removing Vert.x from the DiscV4 UDP layer. It also hardens ENR reuse logic in NodeRecordManager and updates DiscV4 PING handling for EIP-8 tolerance around malformed to fields, with broad test updates.

Changes:

  • Introduces V4Transport and a Netty UDP transport (NettyV4Transport), wiring a new NettyPeerDiscoveryAgent via the factory paths.
  • Refactors PeerDiscoveryAgentV4 to use the transport abstraction and a single-thread dispatch executor for ordered controller mutations.
  • Fixes ENR “reuse-if-unchanged” equality checks (compressed pubkey + fork-id wrapper shape) and updates PING packet to handling to be optional for EIP-8 compliance.

Reviewed changes

Copilot reviewed 37 out of 37 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
ethereum/p2p/src/test/java/org/hyperledger/besu/ethereum/p2p/network/DefaultP2PNetworkTestBuilder.java Removes Vert.x wiring from discovery factory usage in tests.
ethereum/p2p/src/test/java/org/hyperledger/besu/ethereum/p2p/discovery/discv4/transport/NettyV4TransportTest.java Adds transport-level UDP bind/send/stop tests for Netty transport.
ethereum/p2p/src/test/java/org/hyperledger/besu/ethereum/p2p/discovery/discv4/PeerDiscoveryBootstrappingTest.java Updates assertions for PingPacketData.getTo() becoming optional.
ethereum/p2p/src/test/java/org/hyperledger/besu/ethereum/p2p/discovery/discv4/PeerDiscoveryAgentV4Test.java Updates ENR reuse assertions and packet serializer API change (Bytes.size()).
ethereum/p2p/src/test/java/org/hyperledger/besu/ethereum/p2p/discovery/discv4/NettyPeerDiscoveryAgentTest.java Adds focused tests for new Netty agent send/error-stop gating behavior.
ethereum/p2p/src/test/java/org/hyperledger/besu/ethereum/p2p/discovery/discv4/internal/ScheduledExecutorTimerUtilTest.java Adds tests for JDK scheduled-executor-backed timer util.
ethereum/p2p/src/test/java/org/hyperledger/besu/ethereum/p2p/discovery/discv4/internal/PeerDiscoveryControllerTest.java Updates mocks/matchers for optional PING to.
ethereum/p2p/src/test/java/org/hyperledger/besu/ethereum/p2p/discovery/discv4/internal/packet/ping/PingPacketDataRlpWriterTest.java Updates construction for optional to.
ethereum/p2p/src/test/java/org/hyperledger/besu/ethereum/p2p/discovery/discv4/internal/packet/ping/PingPacketDataRlpReaderTest.java Updates reader construction and assertions for optional to.
ethereum/p2p/src/test/java/org/hyperledger/besu/ethereum/p2p/discovery/discv4/internal/packet/ping/PingPacketDataFactoryTest.java Updates factory expectations for optional to.
ethereum/p2p/src/test/java/org/hyperledger/besu/ethereum/p2p/discovery/discv4/internal/packet/PacketSerializerTest.java Updates serializer return type from Vert.x Buffer to Bytes.
ethereum/p2p/src/test/java/org/hyperledger/besu/ethereum/p2p/discovery/discv4/internal/packet/PacketDeserializerTest.java Updates deserializer input type from Vert.x Buffer to Bytes.
ethereum/p2p/src/test/java/org/hyperledger/besu/ethereum/p2p/discovery/discv4/internal/MockPeerDiscoveryAgent.java Refactors mock agent to route through V4Transport and new dispatch executor model.
ethereum/p2p/src/main/java/org/hyperledger/besu/ethereum/p2p/discovery/PeerDiscoveryAgent.java Adds prepareHandlers() hook for pre-bind handler registration.
ethereum/p2p/src/main/java/org/hyperledger/besu/ethereum/p2p/discovery/NodeRecordManager.java Fixes ENR reuse equality checks (compressed pubkey + wrapped fork-id field).
ethereum/p2p/src/main/java/org/hyperledger/besu/ethereum/p2p/discovery/discv4/VertxPeerDiscoveryAgent.java Removes Vert.x-based DiscV4 UDP implementation (deleted).
ethereum/p2p/src/main/java/org/hyperledger/besu/ethereum/p2p/discovery/discv4/V4Transport.java Introduces transport abstraction interface for DiscV4 UDP send/receive.
ethereum/p2p/src/main/java/org/hyperledger/besu/ethereum/p2p/discovery/discv4/transport/NettyV4Transport.java Adds Netty NIO UDP channel transport implementation (IPv4-only).
ethereum/p2p/src/main/java/org/hyperledger/besu/ethereum/p2p/discovery/discv4/PeerDiscoveryAgentV4.java Ports agent to V4Transport, adds ordered dispatch executor and raw inbound decode path.
ethereum/p2p/src/main/java/org/hyperledger/besu/ethereum/p2p/discovery/discv4/PeerDiscoveryAgentFactoryV4.java Switches factory wiring from Vert.x agent to Netty agent + Netty transport.
ethereum/p2p/src/main/java/org/hyperledger/besu/ethereum/p2p/discovery/discv4/NettyPeerDiscoveryAgent.java Adds Netty-backed DiscV4 agent with scheduled-executor timer + worker pool.
ethereum/p2p/src/main/java/org/hyperledger/besu/ethereum/p2p/discovery/discv4/internal/VertxTimerUtil.java Removes Vert.x timer util (deleted).
ethereum/p2p/src/main/java/org/hyperledger/besu/ethereum/p2p/discovery/discv4/internal/ScheduledExecutorTimerUtil.java Adds scheduled-executor-based timer util implementation.
ethereum/p2p/src/main/java/org/hyperledger/besu/ethereum/p2p/discovery/discv4/internal/ScheduledExecutorAsyncExecutor.java Adds executor-service-based async worker wrapper for controller packet creation.
ethereum/p2p/src/main/java/org/hyperledger/besu/ethereum/p2p/discovery/discv4/internal/packet/ping/PingPacketDataRlpWriter.java Updates writer to encode optional to (currently assumes present for outbound).
ethereum/p2p/src/main/java/org/hyperledger/besu/ethereum/p2p/discovery/discv4/internal/packet/ping/PingPacketDataRlpReader.java Updates reader to produce PingPacketData with optional to for malformed tolerance.
ethereum/p2p/src/main/java/org/hyperledger/besu/ethereum/p2p/discovery/discv4/internal/packet/ping/PingPacketDataFactory.java Updates factory to wrap to in Optional.
ethereum/p2p/src/main/java/org/hyperledger/besu/ethereum/p2p/discovery/discv4/internal/packet/ping/PingPacketData.java Makes to optional and updates string representation accordingly.
ethereum/p2p/src/main/java/org/hyperledger/besu/ethereum/p2p/discovery/discv4/internal/packet/PacketSerializer.java Switches serializer output from Vert.x Buffer to Bytes.
ethereum/p2p/src/main/java/org/hyperledger/besu/ethereum/p2p/discovery/discv4/internal/packet/PacketDeserializer.java Switches deserializer input from Vert.x Buffer to Bytes.
ethereum/p2p/src/main/java/org/hyperledger/besu/ethereum/p2p/discovery/discv4/Endpoint.java Improves standalone endpoint decode list-leave behavior on decode errors.
ethereum/p2p/src/main/java/org/hyperledger/besu/ethereum/p2p/discovery/DefaultPeerDiscoveryAgentFactory.java Removes Vert.x from default factory builder/wiring.
ethereum/p2p/build.gradle Adds Netty handler/transport dependencies for new UDP transport/agent.
ethereum/eth/src/test/java/org/hyperledger/besu/ethereum/eth/transactions/TestNode.java Removes Vert.x dependency from discovery factory usage in tests.
ethereum/api/src/test/java/org/hyperledger/besu/ethereum/api/jsonrpc/JsonRpcHttpServiceRpcApisTest.java Removes Vert.x dependency from discovery factory usage in tests.
app/src/test/java/org/hyperledger/besu/RunnerBuilderTest.java Updates test setup for ENR reuse behavior by configuring fork blocks.
app/src/main/java/org/hyperledger/besu/RunnerBuilder.java Removes Vert.x dependency from discovery agent factory wiring.

- Endpoint.maybeDecodeStandalone: catch any RuntimeException, not just
  IllegalPortException, so malformed "to" fields don't abort PING
  processing (EIP-8). decodeStandalone now skips to the end of a
  malformed endpoint list before leaving it, keeping the RLP cursor in
  sync with the enclosing list regardless of where decoding failed.
- ScheduledExecutorAsyncExecutor: complete the future exceptionally
  instead of throwing synchronously when submit() is rejected
  (e.g. during executor shutdown).
- ScheduledExecutorTimerUtil/TimerUtil: catch exceptions from periodic
  handlers so one failure doesn't permanently cancel the scheduled
  task, and name each periodic timer for clearer error logs.

Addresses Copilot review comments on PR besu-eth#10716.

Signed-off-by: Usman Saleem <usman@usmans.info>
…tty migration

- PingPacketDataRlpReader: restore inbound PING expiry validation
  (dropped when the reader bypassed PingPacketDataFactory to tolerate
  malformed "to" fields per EIP-8).
- NettyPeerDiscoveryAgent.stop(): run controller.stop() on the single
  dispatch thread instead of whichever Netty thread completes
  transport.stop(), closing a shutdown-time race with in-flight
  timer/dispatch work.
- PeerDiscoveryController.createPacket(): route the post-signing
  continuation through the same dispatch executor used for inbound
  packet matching, closing a race on PeerInteractionState's filter.
- PeerDiscoveryAgentV4/NettyPeerDiscoveryAgent/MockPeerDiscoveryAgent:
  add a dedicated single-thread decode executor, separate from the
  signing pool, restoring the inbound packet arrival-order guarantee
  Vert.x provided via an ordered executor.
- NettyV4Transport: wrap UDP bind failures in PeerDiscoveryServiceException
  with a descriptive message again, and restore the DEBUG/ERROR
  severity split in the inbound exception handler.
- NettyV4Transport/NettyPeerDiscoveryAgent: create executors/event-loop
  group lazily in start() instead of eagerly in the constructor, so
  discovery-disabled nodes don't carry idle threads.
- MockPeerDiscoveryAgent: exclude stopped agents from the mock
  transport's address-based routing match, and replace the
  constructor-chain/mutable-array back-reference hack with a
  post-construction setter mirroring V4Transport.setInboundHandler.

Fixes 9 findings from a full review of PR besu-eth#10716.

Signed-off-by: Usman Saleem <usman@usmans.info>
Netty's parameterless shutdownGracefully() defaults to a 2s quiet
period (plus a 15s timeout), unconditionally waited out even when the
event loop is already idle. Confirmed via a live mainnet run that this
accounted for the entire ~2.1s delay observed during node shutdown
(NettyV4Transport's event-loop-group teardown, not RLPx/eth-subprotocol
as initially suspected). Nothing is legitimately in flight by the time
this transport shuts down its dedicated single-thread event loop, so
use an explicit 0ms quiet period / 500ms timeout instead.

Also add INFO-level lifecycle logging around NettyPeerDiscoveryAgent
and NettyV4Transport start/stop, matching the pattern already used by
other components (Synchronizer, NetworkRunner, EthProtocolManager) —
this is what surfaced the timing data above.

Signed-off-by: Usman Saleem <usman@usmans.info>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 41 out of 41 changed files in this pull request and generated 3 comments.

- NettyPeerDiscoveryAgent: give each crypto-pool thread a unique name
  (discv4-crypto-0/1) instead of both sharing "discv4-crypto", making
  thread dumps and logs unambiguous.
- NettyV4Transport: only install the TRACE LoggingHandler on the UDP
  pipeline when TRACE is actually enabled for it, avoiding per-packet
  overhead when it's off.
- PeerDiscoveryController: update a stale comment in createPacket()
  that still referenced the "vertx event thread"; this path is now
  Vert.x-free and the real concern is the discovery dispatch/timer
  thread.

Signed-off-by: Usman Saleem <usman@usmans.info>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 41 out of 41 changed files in this pull request and generated no new comments.

Signed-off-by: Usman Saleem <usman@usmans.info>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants