Skip to content

Commit e397a98

Browse files
docs: add flaky test quarantine and remediation policy (#43)
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent 3c1268e commit e397a98

1 file changed

Lines changed: 66 additions & 0 deletions

File tree

CONTRIBUTING.md

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
# Contributing
2+
3+
Thanks for considering contributing to this project. Ways you can help:
4+
5+
- [Create a pull request](https://help.github.com/articles/creating-a-pull-request)
6+
- [Add an issue](https://github.com/braintree/wrap-promise/issues)
7+
8+
## Development
9+
10+
Clone this repo, then install the project's dependencies:
11+
12+
```
13+
npm install
14+
```
15+
16+
## Tests
17+
18+
```
19+
npm test
20+
```
21+
22+
### Test Stability, Quarantine & Remediation Policy
23+
24+
Flaky tests are any test that passes and fails intermittently without code changes (this could be due to things like timeouts, inconsistent return values, poor assertions, etc). Flaky tests erode confidence in the test suite and slow down development. This section defines how we detect, quarantine, and remediate them.
25+
26+
#### Detection
27+
28+
- **Automatic retries**: Jest unit tests can be configured to automatically retry on failure via `jest.retryTimes()`. Playwright integration tests (where applicable) have retries configured per-project. A test that fails once but passes on retry can be considered as potentially flaky.
29+
- **Repeat failures**: If a test fails intermittently across multiple CI runs or PRs, it should be reported by opening a GitHub issue with the `flaky-test` label. Include the test name, file path, failure frequency, and any relevant error output.
30+
31+
#### Quarantine
32+
33+
When a test is confirmed flaky:
34+
35+
1. **Open a tracking issue** in GitHub Issues. Use the `flaky-test` label and include:
36+
- Test file path and test name
37+
- Observed failure rate and pattern (e.g., "fails ~10% of runs", "only in CI")
38+
- Stack trace or error message
39+
2. **Skip the test** using `.skip` and add a comment referencing the tracking issue:
40+
41+
```javascript
42+
// Quarantined: flaky due to timing sensitivity (https://github.com/braintree/wrap-promise/issues/123)
43+
it.skip("should handle the edge case", function () {
44+
// ...
45+
});
46+
```
47+
48+
For `describe` blocks with multiple flaky tests, you may quarantine the entire block:
49+
50+
```javascript
51+
// Quarantined: intermittent failures under load (https://github.com/braintree/wrap-promise/issues/123)
52+
describe.skip("feature under investigation", function () {
53+
// ...
54+
});
55+
```
56+
57+
3. **Skipped tests are excluded from coverage reporting.** Since quarantined tests do not execute, their associated source code is not counted toward coverage metrics. This is expected and acceptable while the test is in quarantine.
58+
59+
#### Remediation SLA
60+
61+
Quarantined tests must be remedied within **30 days** of being quarantined. Remediation means one of the following:
62+
63+
- **Fixed**: The root cause is identified and corrected, the `.skip` is removed, and the test passes reliably.
64+
- **Deleted**: If the test is no longer valuable or the feature it covers has changed, delete the test and close the tracking issue.
65+
66+
If a quarantined test exceeds **14 days** without progress, it should be escalated and prioritized. Tests that remain skipped indefinitely are not acceptable.

0 commit comments

Comments
 (0)