Improve KinD E2E failure diagnostics

[kahirokunn]

Why

KinD E2E failures were difficult to diagnose because test namespaces could be deleted before postmortem collection, one failed kubectl command could stop collection, and logs were not uploaded as bounded artifacts.

Proposed Changes

• keep failed test namespaces available for postmortem collection
• save bounded cluster state and current/previous container logs without dumping Secrets or Pod environment values
• retain failure diagnostics as artifacts for seven days

Release Note

NONE

[knative-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: kahirokunn
Once this PR has been reviewed and has the lgtm label, please assign dsimansk for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[kahirokunn]

/ok-to-test

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 43.57%. Comparing base (2c86680) to head (4554fb7).

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #800      +/-   ##
==========================================
+ Coverage   43.50%   43.57%   +0.07%     
==========================================
  Files          68       68              
  Lines        4259     4259              
==========================================
+ Hits         1853     1856       +3     
+ Misses       2282     2278       -4     
- Partials      124      125       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[kahirokunn]

@astelmashenko Hi ✋ PTAL. Thank you 🙏

[astelmashenko]

@kahirokunn , shell script is hard too read and has custom functions, it is always hard to maintain shell scripts.
Current approach was the same across projects, e.g. rabbitmq broker https://github.com/knative-extensions/eventing-rabbitmq/blob/main/.github/workflows/kind-e2e.yaml

I'd also doublecheck reusable actions if there is something to reuse https://github.com/knative/actions/tree/main/.github/workflows

retain failure diagnostics as artifacts for seven days

I'm not sure if I can decide such things. I support codebase. Regarding infrastructure things it is better to ask @Cali0707, @dprotaso or @pierDipi

[kahirokunn]

@kahirokunn , shell script is hard too read and has custom functions, it is always hard to maintain shell scripts. Current approach was the same across projects, e.g. rabbitmq broker https://github.com/knative-extensions/eventing-rabbitmq/blob/main/.github/workflows/kind-e2e.yaml

I'd also doublecheck reusable actions if there is something to reuse https://github.com/knative/actions/tree/main/.github/workflows

retain failure diagnostics as artifacts for seven days

I'm not sure if I can decide such things. I support codebase. Regarding infrastructure things it is better to ask @Cali0707, @dprotaso or @pierDipi

Thanks for the feedback. I originally based this on the approach used by eventing-kafka-broker:

• managed test environments: https://github.com/knative-extensions/eventing-kafka-broker/blob/main/test/e2e_new/broker_test.go#L48-L62
• saving logs under ARTIFACTS: https://github.com/knative-extensions/eventing-kafka-broker/blob/main/test/e2e-common.sh#L374-L383
  I also checked knative/actions, but couldn’t find a reusable workflow for collecting KinD failure diagnostics.

WDYT?

[Cali0707]

I can take a look at this next week

/assign @Cali0707