ci: fail payload aggregation loudly when total is zero

Parsing (TRX regex or the agent-log regex in AgentLogFile) breaking would
otherwise yield a valid-but-empty result and silently report zeros to New
Relic with a green build. Add build/Scripts/check-payload-not-empty.sh (with
tests) and a non-continue-on-error guard step before the report step that
fails the aggregation job when the consolidated total is <= 0.

Author: tippmar-nr

.github/workflows/all_solutions.yml

@@ -1352,6 +1352,11 @@ jobs:
           path: final_payload_summary.json
           if-no-files-found: warn
 
+      - name: Guard against empty payload data
+        if: inputs.aggregate_payload_data == true || github.event_name == 'schedule'
+        run: bash build/Scripts/check-payload-not-empty.sh final_payload_summary.json
+        shell: bash
+
       - name: Report payload metrics to New Relic (staging)
         if: inputs.aggregate_payload_data == true || github.event_name == 'schedule'
         continue-on-error: true

build/Scripts/check-payload-not-empty.sh

@@ -0,0 +1,48 @@
+#!/usr/bin/env bash
+# Guard: fail loudly if the consolidated payload summary reports (near-)zero total bytes.
+#
+# When payload aggregation runs, the test suites always send a large amount of
+# data (every test app at least does a connect handshake), so an overall total
+# of zero is never legitimate - it means the TRX / agent-log parsing broke
+# (e.g. a log-line format change) somewhere upstream. Without this guard the
+# pipeline would silently report zeros to New Relic and the dashboard would
+# flatline with no failure. This step turns that silent failure into a loud one.
+#
+# Usage:
+#   check-payload-not-empty.sh [SUMMARY_FILE]
+#   SUMMARY_FILE            path to final_payload_summary.json
+#                           (arg 1, or PAYLOAD_SUMMARY_FILE env var)
+#   MIN_EXPECTED_PAYLOAD_BYTES  floor; fail if total <= this (default 0).
+#                               Raise it later to catch partial breakage too.
+#
+# Exit codes:
+#   0  total is above the floor
+#   1  summary file missing/unparseable, or total <= floor
+set -euo pipefail
+
+SUMMARY_FILE="${1:-${PAYLOAD_SUMMARY_FILE:-}}"
+FLOOR="${MIN_EXPECTED_PAYLOAD_BYTES:-0}"
+
+if [ -z "$SUMMARY_FILE" ] || [ ! -f "$SUMMARY_FILE" ]; then
+  echo "ERROR: payload summary file not found (got '$SUMMARY_FILE')." >&2
+  exit 1
+fi
+
+total="$(jq -r '(.bytes // 0) | floor' "$SUMMARY_FILE" 2>/dev/null || true)"
+case "$total" in
+  '' | *[!0-9]*)
+    echo "ERROR: could not read a numeric total from '$SUMMARY_FILE' (got '$total')." >&2
+    exit 1
+    ;;
+esac
+
+echo "Overall payload bytes: $total (floor: $FLOOR)"
+if [ "$total" -le "$FLOOR" ]; then
+  echo "ERROR: payload total ($total) is at or below the floor ($FLOOR)." >&2
+  echo "This almost certainly means the TRX / agent-log parsing broke (e.g. a log-line" >&2
+  echo "format change), not a real zero. Failing so it is investigated instead of" >&2
+  echo "silently reporting zeros to New Relic." >&2
+  exit 1
+fi
+
+echo "Payload guard passed."

build/Scripts/tests/test-check-payload-not-empty.sh

@@ -0,0 +1,36 @@
+#!/usr/bin/env bash
+# Tests for check-payload-not-empty.sh: the guard must pass on a real total and
+# fail on zero / missing bytes / missing file / below-floor.
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+SCRIPT="$SCRIPT_DIR/../check-payload-not-empty.sh"
+tmp="$(mktemp -d)"
+trap 'rm -rf "$tmp"' EXIT
+
+fail() { echo "FAIL: $1" >&2; exit 1; }
+
+# Non-zero total -> pass (exit 0).
+echo '{"bytes": 69209939, "byType": {}}' > "$tmp/ok.json"
+bash "$SCRIPT" "$tmp/ok.json" >/dev/null 2>&1 || fail "expected pass on non-zero total"
+
+# Zero total -> fail (exit 1).
+echo '{"bytes": 0, "byType": {}}' > "$tmp/zero.json"
+if bash "$SCRIPT" "$tmp/zero.json" >/dev/null 2>&1; then fail "expected failure on zero total"; fi
+
+# Missing bytes field (treated as 0) -> fail.
+echo '{"byType": {}}' > "$tmp/missing.json"
+if bash "$SCRIPT" "$tmp/missing.json" >/dev/null 2>&1; then fail "expected failure on missing bytes field"; fi
+
+# Missing file -> fail.
+if bash "$SCRIPT" "$tmp/does-not-exist.json" >/dev/null 2>&1; then fail "expected failure on missing file"; fi
+
+# Below an explicit floor -> fail.
+echo '{"bytes": 500}' > "$tmp/low.json"
+if MIN_EXPECTED_PAYLOAD_BYTES=1000 bash "$SCRIPT" "$tmp/low.json" >/dev/null 2>&1; then fail "expected failure below floor"; fi
+
+# Above an explicit floor -> pass.
+echo '{"bytes": 5000}' > "$tmp/high.json"
+MIN_EXPECTED_PAYLOAD_BYTES=1000 bash "$SCRIPT" "$tmp/high.json" >/dev/null 2>&1 || fail "expected pass above floor"
+
+echo "PASS: all guard assertions passed"