fix(parity): bound the external validators so a stall fails instead of hanging

A Pipeline Parity hl7 (ips) job sat for 30+ minutes on a single line of output and
had to be cancelled by hand. Three compounding gaps, none of them in the
validators themselves:

1. spawnAndCapture settled only on the child's close/error. The validators print
   nothing until they finish, so a subprocess blocked on the network was
   indistinguishable from one doing work, and the promise never settled. It now
   kills the child after a wall-clock bound and rejects with the byte count and
   stderr tail, so the failure names itself. Override with
   PARITY_VALIDATOR_TIMEOUT_MS; default 15 minutes, against single-digit minutes
   for a whole package.

2. No job in _parity-tests.yml declared timeout-minutes, so each inherited
   GitHub's 6-hour default. Without the manual cancel that job would have burned
   six hours. internal gets 45 (it generates what the others reuse), firely and
   hl7 get 30.

3. downloadValidator handled only 'error' on its release download, so a stalled
   socket waited forever too. Now setTimeout + destroy at 120s.

Verified by reproducing the failure rather than reasoning about it: runHL7Batch
pointed at a stand-in batch-validate.js that sleeps forever and prints nothing —
the real shape of the stuck job — rejects after exactly the configured bound with
"produced no result within 3s and was killed. Captured 0 bytes of stdout".

Not the cause, though both were suspects: fhirpath 5.1.0 introduced a one-hour
pending-request timeout, but Phase 3 runs the external HL7 Java validator and
fhirpath only drives our internal ones. And the job logged SKIP_GENERATION with
resources loaded from a manifest, so no emitter ran in it at all.

Still open, seen in the same log and not addressed here: while validating ips, the
exclusion line listed 27 classes belonging to other packages (AUCore*, ISiK*,
KBV*, USCoreVitalSigns*). Either a global summary printed once or
getExcludedFilePaths is not filtered per package.

Author: quotentiroler

.github/workflows/_parity-tests.yml

@@ -113,6 +113,9 @@ jobs:
   internal:
     needs: [setup]
     runs-on: ubuntu-latest
+    # Generates the packages the other two jobs reuse, so it is the longest of the
+    # three; bounded for the same reason as hl7 below.
+    timeout-minutes: 45
     permissions:
       contents: read
     strategy:
@@ -239,6 +242,9 @@ jobs:
     needs: [setup, internal, resolve-versions]
     if: ${{ !cancelled() }}
     runs-on: ubuntu-latest
+    # Same exposure as hl7: an external validator subprocess that can stall on the
+    # network with no output.
+    timeout-minutes: 30
     permissions:
       contents: read
     strategy:
@@ -368,6 +374,11 @@ jobs:
     needs: [setup, internal, resolve-versions]
     if: ${{ !cancelled() }}
     runs-on: ubuntu-latest
+    # The external validator runs as a subprocess with no output while it works, so
+    # a network stall inside it reads as a live job. Without a bound it inherits
+    # GitHub's 6-hour default: one hl7 (ips) job sat for 30+ minutes on nothing
+    # before being cancelled by hand. A whole package takes single-digit minutes.
+    timeout-minutes: 30
     permissions:
       contents: read
     strategy:

scripts/hl7-validator/batch-validate.js

@@ -115,8 +115,12 @@ async function downloadValidator(version) {
   return new Promise((resolve, reject) => {
     const file = createWriteStream(jarPath);
 
+    // A stalled socket had no bound: only 'error' was handled, so a hung release
+    // download waited forever and the whole parity job waited with it.
+    const DOWNLOAD_TIMEOUT_MS = 120_000;
+
     const download = (downloadUrl) => {
-      httpsGet(downloadUrl, (response) => {
+      const request = httpsGet(downloadUrl, (response) => {
         if (response.statusCode === 302 || response.statusCode === 301) {
           // Follow redirect
           download(response.headers.location);
@@ -135,6 +139,10 @@ async function downloadValidator(version) {
       }).on('error', (err) => {
         reject(err);
       });
+      request.setTimeout(DOWNLOAD_TIMEOUT_MS, () => {
+        request.destroy();
+        reject(new Error(`Validator download stalled for ${DOWNLOAD_TIMEOUT_MS / 1000}s: ${downloadUrl}`));
+      });
     };
 
     download(url);

src/test/parity/validatorAdapters.ts

@@ -65,6 +65,19 @@ function parseValidatorJson(jsonOutput: string, label: string): Map<string, Vali
 }
 
 /** Spawn a child process, capture stdout/stderr, return stdout on close. */
+/**
+ * Wall-clock bound on an external validator.
+ *
+ * The validators emit nothing until they finish, so a stall inside one is
+ * indistinguishable from work in progress. Settling only on `close`/`error` meant
+ * a validator blocked on the network hung the run indefinitely: one hl7 (ips) job
+ * sat for 30+ minutes on a single line of output before being cancelled by hand,
+ * and the workflow's jobs had no timeout to stop it either.
+ *
+ * A whole package is single-digit minutes, so this is generous.
+ */
+const VALIDATOR_TIMEOUT_MS = Number(process.env.PARITY_VALIDATOR_TIMEOUT_MS ?? 15 * 60 * 1000);
+
 function spawnAndCapture(
   cmd: string,
   args: string[],
@@ -78,17 +91,31 @@ function spawnAndCapture(
     });
     let out = '';
     let err = '';
+    let settled = false;
+    const finish = (fn: () => void) => { if (!settled) { settled = true; clearTimeout(timer); fn(); } };
+
+    // unref() so a pending timer cannot itself hold the process open.
+    const timer = setTimeout(() => {
+      child.kill('SIGKILL');
+      const errTail = err.length > 500 ? `...${err.slice(-500)}` : err;
+      finish(() => reject(new Error(
+        `${opts.label} produced no result within ${Math.round(VALIDATOR_TIMEOUT_MS / 1000)}s and was killed. `
+        + `Captured ${out.length} bytes of stdout. stderr: ${errTail || '(none)'}`,
+      )));
+    }, VALIDATOR_TIMEOUT_MS);
+    timer.unref?.();
+
     child.stdout.on('data', d => { out += d.toString(); });
     child.stderr.on('data', d => { err += d.toString(); });
-    child.on('error', reject);
+    child.on('error', e => finish(() => reject(e)));
     child.on('close', (code) => {
       if (err && opts.verbose) console.log(`${opts.label} stderr:`, err);
       if (code !== 0 && !out.trim()) {
         const errTail = err.length > 500 ? `...${err.slice(-500)}` : err;
-        reject(new Error(`${opts.label} exited with code ${code}. stderr: ${errTail}`));
+        finish(() => reject(new Error(`${opts.label} exited with code ${code}. stderr: ${errTail}`)));
         return;
       }
-      resolve(out);
+      finish(() => resolve(out));
     });
   });
 }