Skip to content

Document Test-Driven Workflow in CONTRIBUTING.md; Document PR audit file requirement #2152

Description

@oberstet

Summary

The current CONTRIBUTING.md has a basic "Contributing Code" section but lacks detailed guidance on the test-driven bug fix / feature workflow that we expect contributors to follow. This workflow ensures PRs are convincing beyond any doubt and prevents regressions.

Problem

Contributors (especially new ones) often submit PRs that:

  • Don't include tests proving the bug exists
  • Don't demonstrate the fix actually works
  • Don't provide evidence from both local runs AND CI

This makes review harder and less certain.

Proposed Documentation

Add a new section "Bug Fix Workflow" to CONTRIBUTING.md documenting the expected process:

Bug Fix Workflow (Test-Driven)

When fixing a bug, follow this workflow to create a convincing, reviewable PR:

1. File Issue & Discuss

  • File a GitHub issue describing the bug
  • Discuss and gather agreement: "yes, current behavior is wrong"
  • Reference any related issues or user reports

2. Create PR with Failing Test

  • Create a feature branch
  • Write a test that demonstrates the bug (test should FAIL)
  • Include any audit/fixture files needed
  • Push the PR

3. Prove the Bug Exists

  • Run tests locally → verify they FAIL
  • Let GitHub Actions run → verify CI FAILS
  • Leave a PR comment with:
    • Your local test output showing failure
    • Link to the failing CI workflow run

4. Implement the Fix

  • Fix the bug
  • Push the fix to the same PR

5. Prove the Fix Works

  • Run tests locally → verify they PASS
  • Let GitHub Actions run → verify CI PASSES
  • Leave a PR comment with:
    • Your local test output showing success
    • Link to the passing CI workflow run

6. Request Review

  • Comment: "Ready for review & merge!"
  • Ping maintainers if needed

Why This Workflow?

This leaves maintainers with a completely convincing PR because:

  • There is a behavior considered incorrect (bug)
  • The test proves the current implementation is behaving incorrectly
  • The fix proves the new implementation does fix it
  • The test coverage ensures we won't regress later

Example PR Comments

After step 3 (proving bug exists):

## Bug Confirmed

Local test run:
​```
$ just test
...
FAILED test_foo.py::test_bar - AssertionError: expected X, got Y
​```

CI also fails: https://github.com/crossbario/crossbar/actions/runs/12345

After step 5 (proving fix works):

## Fix Verified

Local test run:
​```
$ just test
...
PASSED test_foo.py::test_bar
​```

CI passes: https://github.com/crossbario/crossbar/actions/runs/12346

Ready for review & merge!

Scope

This documentation should be added to CONTRIBUTING.md in all Crossbar.io related Python repos: txaio, autobahn-python, zlmdb, cfxdb, wamp-xbr

Related

Checklist

  • I have searched existing issues to avoid duplicates
  • I have provided a minimal reproducible example
  • I have included version information
  • I have included error messages/logs

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions