pySCG: add testing framework for code examples and docs#1011
pySCG: add testing framework for code examples and docs#1011
Conversation
…d docs Add pytest-based testing framework with 765 test cases to validate Python code examples and Markdown documentation structure. Features: - Python validation: syntax, deprecation warnings, expected output - Markdown validation: required sections, tables, section order, code refs - Link validation: internal links, index table, code file references - Smart CI/CD: tests only changed CWE directories on PRs (~1 min) - Full suite: runs all tests on pushes to main (~3 min) - Multi-version: Python 3.9-3.14 testing with tox - Clean reporting: generate_issue_report.py for actionable summaries - Auto cleanup: removes test artifacts after test runs CI/CD Integration: - GitHub Actions workflow with matrix strategy - Lychee link checker (internal links in CI, external available locally) - Coverage reporting to Codecov - Fast execution with uv package manager Documentation: - Comprehensive tests/README.md with usage examples - Updated CONTRIBUTING.md with testing workflow - KNOWN_ISSUES.md documents 44 files with pre-existing issues The framework identifies issues in 44 files (31 documentation files with missing sections/tables/order issues, 13 Python files with timeouts/warnings) documented in KNOWN_ISSUES.md. These pre-existing issues can be addressed in follow-up PRs. All new contributions will be validated automatically. Closes ossf#1010 Signed-off-by: tommcd <tcmcdermott@gmail.com>
- Add blank lines around lists, headings, and code blocks (MD022, MD031, MD032) - Standardize list style to dashes (MD004) - Add language specifications to code blocks (MD040) - Remove trailing spaces (MD009) - Add trailing newline to KNOWN_ISSUES.md (MD047) - Standardize bold style to underscores (MD050) These errors were caught by GitHub Super-Linter but not by local auto-fix. All 7 files now pass markdownlint validation. Signed-off-by: tommcd <tcmcdermott@gmail.com>
59d5ee7 to
32ebdf7
Compare
Change readme reference from 'README.md' to 'tests/README.md' to match actual file location and fix build error in CI. Signed-off-by: tommcd <tcmcdermott@gmail.com>
Fix bash script that generates pytest -k filter to properly handle single directory case. Trim trailing spaces before joining with ' or ' to avoid invalid pytest expressions like 'CWE-703 or '. Signed-off-by: tommcd <tcmcdermott@gmail.com>
Mark CWE-390 files with EXPECTED_TIMEOUT to skip timeout tests. These files intentionally contain infinite loops/blocking operations for educational purposes as documented in KNOWN_ISSUES.md. Signed-off-by: tommcd <tcmcdermott@gmail.com>
Fix typo 'exammple01.py' -> 'example01.py' in markdown link. Signed-off-by: tommcd <tcmcdermott@gmail.com>
Mark CWE-798/compliant01.py with EXPECTED_FAILURE for deprecated Path() kwargs usage. This is a known issue documented in KNOWN_ISSUES.md that will be addressed in a follow-up PR. Signed-off-by: tommcd <tcmcdermott@gmail.com>
Welcome to Codecov 🎉Once you merge this PR into your default branch, you're all set! Codecov will compare coverage reports and display results in all future pull requests. Thanks for integrating Codecov - We've got you covered ☂️ |
Remove invalid 'exist_ok=True' parameter from Path() constructor. This parameter belongs to Path.mkdir(), not Path(). The invalid usage triggers DeprecationWarning in Python 3.12+ and will cause errors in Python 3.14+. The file is created by open() on the next line, so exist_ok was never needed here. This is a bug fix with no behavior change. Signed-off-by: tommcd <tcmcdermott@gmail.com>
Add critical test to ensure README inlined code blocks match actual Python files (after stripping SPDX headers and test markers). This test catches a common documentation error where: - Code is updated but README isn't updated - SPDX headers are incorrectly included in inlined code - Test markers (EXPECTED_TIMEOUT/EXPECTED_FAILURE) appear in docs Currently identifies 18 files with mismatches (pre-existing issues). These should be fixed in follow-up PRs. Signed-off-by: tommcd <tcmcdermott@gmail.com>
Update generate_issue_report.py to detect and report inlined code mismatches. Regenerated KNOWN_ISSUES.md showing 18 files where README inlined code doesn't match actual Python files. Signed-off-by: tommcd <tcmcdermott@gmail.com>
Fix markdown linting error MD047. Signed-off-by: tommcd <tcmcdermott@gmail.com>
Fix MD012 markdown linting error. Signed-off-by: tommcd <tcmcdermott@gmail.com>
Fix generate_issue_report.py to add trailing newline when saving. Signed-off-by: tommcd <tcmcdermott@gmail.com>
|
Great to see effort invested in cleaning up the codebase and documentation. While we need to do this, I do believe that restructuring and getting the content to a release is more important at the moment. We can surely come up with a solution that does not require Kiro or any specific IDE. To avoid pushing against a GitHub linter we must have a solution matching local vs GitHub runs. Idealistically we have a matching code style support as on option for an IDE. Keeping this simple will avoid barriers for new contributors to join. We can increase complexity after reaching healthy amount of active contributors. Rg Helge |
Add pytest-based testing framework with 783 test cases to validate Python code examples and Markdown documentation structure.
Features:
New: Inlined Code Validation
CI/CD Integration:
Documentation:
The framework identifies issues in 53 files (35 documentation files with missing sections/tables/order/inlined code issues, 11 Python files with timeouts, 1 with deprecation warning) documented in KNOWN_ISSUES.md. These pre-existing issues can be addressed in follow-up PRs. All new contributions will be validated automatically.
Closes #1010