Skip to content

test(e2e): inventory and bound retries for transient downstream failures #9166

Description

@rsliter

Parent: #9159

Problem

E2E recovery behavior is split between broad failed-job reruns and test-specific retry or skip loops. The suite does not have one inventory or one attempt-evidence contract. A later pass can therefore hide the original failure, while deterministic failures can consume more runtime without becoming recoverable.

Scope

  • Inventory every retry, rerun, and transient-skip path in the E2E suite.
  • Record the operation, failure signature, attempt bound, backoff, idempotence basis, owner, result classification, and retained evidence.
  • Fix root causes that NemoClaw owns.
  • Add or tune bounded retries for downstream operations that are idempotent and demonstrably transient.
  • Record external ownership and upstream evidence when an external dependency remains unreliable.
  • Report first-attempt failures, retry attempts, passes after retry, and exhausted retries separately.

Required behavior

  • Product assertions, authentication failures, authorization failures, policy denials, and malformed inputs must not retry.
  • A mutating operation with an ambiguous transport result must stop or reconcile external state before another attempt.
  • An exhausted retry must remain failed and retain its attempt history.
  • Cleanup failures must remain visible.
  • Logs and artifacts must redact credentials and credential-bearing request data.
  • Candidate-controlled artifacts must not authorize a trusted GitHub Actions write.

Acceptance criteria

  • A version-controlled inventory covers every discovered retry, rerun, and transient-skip path.
  • Each eligible current path uses the bounded policy or records why it is not eligible.
  • Per-target and aggregate evidence distinguish each attempt outcome.
  • A pass after retry remains visible as degraded evidence.
  • An exhausted retry identifies the final failure and the external owner when known.
  • Release conclusions and required checks do not change.

Test evidence

  • Deterministic tests cover transient success, transient exhaustion, deterministic failure, ambiguous mutation, cleanup failure, malformed evidence, and credential redaction.
  • Existing live E2E tests provide operational evidence without adding new tests.

Non-goals

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: e2eEnd-to-end tests, nightly failures, or validation infrastructure

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions