Fix `lwd_rpc_send_tx` test failure: "test child was already taken" after 570s timeout

[mpguerra]

Problem

The stateful integration test stateful::lightwalletd::lwd_rpc_send_tx is failing with exit code 100 after running for over 570 seconds. The test logs show:

test stateful::lightwalletd::lwd_rpc_send_tx has been running for over 60 seconds
...
test child was already taken

Log source: GCP CI run #0881709

Root Cause

The error "test child was already taken" occurs in the test cleanup phase (zebrad/tests/common/lightwalletd.rs:622-638) when attempting to kill the zebrad child process. This indicates:

1. Premature process termination: A lightwalletd or zebrad child process exited unexpectedly before cleanup
2. Resource exhaustion: The long-running sync operation (570+ seconds) to mainnet tip (~block 3469492) may have depleted system resources
3. Unsafe child process handling: The code assumes wait_with_output() will return a valid output, but if the child was already consumed or terminated abnormally, it fails

Test Context

• Test type: Stateful lightwalletd RPC integration test
• Network: Mainnet
• Sync progress at timeout: 99.97% (height 3469476 out of 3469495)
• Duration: 570+ seconds (expected for full sync, but cleanup failed)
• Affected code: zebrad/tests/common/lightwalletd.rs:622-638

Recommended Fix

1. Add graceful error handling for child process cleanup

Replace the current cleanup logic that assumes success:

// Current code (lines 622-638):
zebrad.kill(false)?;

if let Some(mut lightwalletd) = lightwalletd {
    lightwalletd.kill(false)?;
    let lightwalletd_output = lightwalletd.wait_with_output()?.assert_failure()?;
    lightwalletd_output.assert_was_killed()
        .wrap_err("Possible port conflict...")?;
}

let zebrad_output = zebrad.wait_with_output()?.assert_failure()?;
zebrad_output.assert_was_killed()
    .wrap_err("Possible port conflict...")?;

With resilient error handling:

// Kill both processes, log failures but continue
if let Err(e) = zebrad.kill(false) {
    tracing::warn!("failed to kill zebrad: {}", e);
}

if let Some(mut lightwalletd) = lightwalletd {
    if let Err(e) = lightwalletd.kill(false) {
        tracing::warn!("failed to kill lightwalletd: {}", e);
    }
    
    match lightwalletd.wait_with_output() {
        Ok(output) => {
            output.assert_failure()
                .or_else(|_| {
                    tracing::warn!("lightwalletd did not exit with failure status");
                    Ok(())
                })?
                .assert_was_killed()
                .wrap_err("lightwalletd was not killed, possible port conflict")?;
        }
        Err(e) if e.to_string().contains("already taken") => {
            tracing::warn!("lightwalletd child already consumed/terminated: {}", e);
        }
        Err(e) => return Err(e).wrap_err("failed to wait on lightwalletd"),
    }
}

match zebrad.wait_with_output() {
    Ok(output) => {
        output.assert_failure()
            .or_else(|_| {
                tracing::warn!("zebrad did not exit with failure status");
                Ok(())
            })?
            .assert_was_killed()
            .wrap_err("zebrad was not killed, possible port conflict")?;
    }
    Err(e) if e.to_string().contains("already taken") => {
        tracing::warn!("zebrad child already consumed/terminated: {}", e);
    }
    Err(e) => return Err(e).wrap_err("failed to wait on zebrad"),
}

2. Increase transaction send timeout

The RPC test for sending transactions may need more time. Consider adding a configurable timeout or increasing the default lightwalletd timeout for transaction operations.

3. Add resource monitoring

Document expected resource usage (disk space, memory) for this test, especially for mainnet full sync. The test currently syncs from near-genesis to chain tip (~3.46M blocks).

Test Coverage

• Verify the test can complete cleanly on isolated runs (no other tests interfering)
• Verify resource cleanup is properly instrumented with logging
• Add a check for OOM conditions and disk space exhaustion

Links

• CI Job: https://github.com/ZcashFoundation/zebra/actions/runs/33644944705/job/100303430541
• Test file: https://github.com/ZcashFoundation/zebra/blob/0881709ff428e2867366957d2748b666501f8eff/zebrad/tests/common/lightwalletd.rs
• Related constants:
  • LIGHTWALLETD_STARTUP_DELAY (line 33)
  • SYNC_FINISHED_REGEX (line 34)

[conradoplg]

My very superficial initial impression is that this a error handling bug for the timeout case, but the test itself should stop timing out with #11378