physrep: skip rtcpu'd / decommissioned hosts when establishing reverse connections

[markhannum]

Summary

The physrep reverse-connection machinery never consulted rtcpu, unlike the normal (non-reverse) source-selection path which already rejects rtcpu'd nodes in physrep_allowed_source(). As a result, a source/intermediary would spawn a reverse-connection worker per target host read from the physrep sources table and spin forever trying to connect to a host that rtcpu reports as down or decommissioned — for example a stale sources-table entry left behind after a machine was decommed — producing an endless stream of "Failed to connect" errors. This change brings the reverse-connection path in line with the rtcpu discipline the normal path already enforces, on both the sending and receiving sides.

Background

For cross-cloud physrep (PPCLD -> PDCLD), the replicant often cannot dial out to its source, so the source dials in to the replicant and hands it a socket (a "reverse connection"), which the replicant then uses to pull logs. There are two sides: the sender of reverse connections is the source/intermediary (db/reverse_conn.c), which runs one worker thread per target host and periodically calls send_reversesql_request(); the receiver is the replicant accepting the inbound reversesql appsock (plugins/reversesql/reversesql.c). Neither side checked rtcpu before this change.

Changes

Sender (db/reverse_conn.c): the reverse-connection worker now skips a connect attempt when the target host is not up per rtcpu, and resumes automatically once the host comes back.

Receiver (plugins/reversesql/reversesql.c): an inbound reversesql request from a source that rtcpu reports as down/decommissioned is now rejected.

A host is considered "not up" when machine_is_up() returns anything other than 1, which covers both rtcpu'd-down (0) and unknown/decommissioned (-1) hosts. Note this is intentionally stricter than physrep_allowed_source(), which uses !nodeup_rtn(...) and would treat an unknown (-1) host as up; the stricter check is what actually catches a fully decommed machine.

The shared predicate physrep_revconn_host_is_up() is exposed via a new revconn_host_is_up <host> message command for testing.

Testing

Adds physrep_revconn_rtcpu to tests/phys_rep_tiered.test, modeled on the existing physrep_rtcpu_source case. Using the fakedr trap to force machine_is_up() to report a host as down, it asserts that revconn_host_is_up returns "up" for a healthy host, "down" once the host is marked rtcpu'd, and "up" again after the mark is cleared. Connections use --admin, since the node under test can itself be the one marked rtcpu'd (drtest) and would otherwise refuse normal client traffic.

Notes

This is framed as hardening rather than the root-cause fix. In the originating incident the SQL engine pool was saturated with inbound comdb2_transaction_logs queries and the timestamps did not correlate with the reverse-connect failures, and the underlying trigger — a decommissioned machine left in the comdb2_physrep_sources table — has been acknowledged and cleaned up separately by the DBA group. This change is defense-in-depth so a stale or decommissioned source entry can no longer cause an endless reverse-connect storm.

[roborivers]

Cbuild submission: Success ✓.
Regression testing: Success ✓.

The first 10 failing tests are:
queuedb_rollover_noroll1_generated **quarantined**
consumer_non_atomic_default_consumer_generated **quarantined**
sc_downgrade [timeout] **quarantined**

[roborivers]

Cbuild submission: Error ⚠.
Regression testing: Success ✓.

The first 10 failing tests are:
ssl_san
sc_parallel_logicalsc_generated
consumer_non_atomic_default_consumer_generated **quarantined**
ssl_set_cmd
ssl_prefer
ssl_dbname
sc_downgrade [timeout] **quarantined**
reco-ddlk-sql [timeout] **quarantined**