ptai 1.1.0 — verification coverage + resilience (Juice Shop 0→12 verified) #280
Workflow file for this run
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| name: pentest-ai | |
| on: | |
| push: | |
| branches: [main] | |
| paths-ignore: | |
| - '**/*.md' | |
| - 'docs/**' | |
| - 'specs/**' | |
| - 'examples/**' | |
| - 'benchmarks/**' | |
| - 'LICENSE' | |
| - '.gitignore' | |
| - '.gitattributes' | |
| - '.dockerignore' | |
| - '.github/CODEOWNERS' | |
| - '.github/ISSUE_TEMPLATE/**' | |
| - 'SECURITY.md' | |
| pull_request: | |
| branches: [main] | |
| paths-ignore: | |
| - '**/*.md' | |
| - 'docs/**' | |
| - 'specs/**' | |
| - 'examples/**' | |
| - 'benchmarks/**' | |
| - 'LICENSE' | |
| - '.gitignore' | |
| - '.gitattributes' | |
| - '.dockerignore' | |
| - '.github/CODEOWNERS' | |
| - '.github/ISSUE_TEMPLATE/**' | |
| - 'SECURITY.md' | |
| # Skip CI for dependabot PRs that only touch lock files (the test job | |
| # already runs against the bumped versions; running again on PR is | |
| # redundant noise). | |
| concurrency: | |
| group: ci-${{ github.ref }} | |
| cancel-in-progress: true | |
| jobs: | |
| # Dedicated job to satisfy the branch-protection rule, which requires a | |
| # status check literally named "pytest (Python 3.12)". This file runs the | |
| # full suite once on ubuntu + Python 3.12; there is no cross-OS/version | |
| # matrix here, so this is the only unguarded pytest invocation to harden. | |
| pytest: | |
| name: pytest (Python 3.12) | |
| runs-on: ubuntu-latest | |
| steps: | |
| - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 | |
| - name: Set up Python 3.12 | |
| uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0 | |
| with: | |
| python-version: "3.12" | |
| cache: pip | |
| - name: Install dependencies | |
| run: | | |
| python -m pip install --upgrade pip | |
| pip install -e ".[dev]" | |
| - name: Run tests | |
| # Engagement-lifecycle E2E is the e2e-juiceshop job's job; ignore here. | |
| # rm -f first so a leftover report.xml from any earlier step can never | |
| # satisfy the guard if this run dies before writing its own. | |
| run: | | |
| rm -f report.xml | |
| pytest tests/ -v --tb=short --junitxml=report.xml --ignore=tests/test_engagement_lifecycle_e2e.py | |
| - name: Verify pytest result is complete (guard against os._exit truncation) | |
| # The load-bearing defense: an os._exit(0) mid-run kills pytest before it | |
| # writes report.xml, so a green return code yields no report at all. The | |
| # guard fails the build on a missing/truncated report or any failure/error. | |
| run: python scripts/check_pytest_result.py report.xml | |
| lint: | |
| name: lint + type check (ubuntu / py3.12) | |
| runs-on: ubuntu-latest | |
| steps: | |
| - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 | |
| - name: Set up Python | |
| uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0 | |
| with: | |
| python-version: "3.12" | |
| cache: pip | |
| - name: Install dependencies | |
| run: | | |
| python -m pip install --upgrade pip | |
| pip install -e ".[dev]" | |
| - name: Run ruff | |
| run: ruff check . | |
| - name: Run mypy | |
| run: mypy mcp_server engine tools agents --ignore-missing-imports | |
| e2e-juiceshop: | |
| name: e2e smoke (juice shop) | |
| runs-on: ubuntu-latest | |
| timeout-minutes: 30 | |
| services: | |
| juiceshop: | |
| # Pinned to v20.0.0 (current bkimminich/juice-shop:latest as of 2026-05-12). | |
| # Unpinned `:latest` periodically pulled new builds that took >60s to | |
| # start, tripping the readiness loop below and red-Xing commits. Bump | |
| # this tag deliberately when re-benchmarking. | |
| image: bkimminich/juice-shop:v20.0.0 | |
| ports: | |
| - 3000:3000 | |
| steps: | |
| - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 | |
| - name: Set up Python | |
| uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0 | |
| with: | |
| python-version: "3.12" | |
| cache: pip | |
| - name: Install ptai + dev deps | |
| run: | | |
| python -m pip install --upgrade pip | |
| pip install -e ".[dev]" | |
| - name: Install minimal external tools (gobuster + ffuf via apt) | |
| run: | | |
| sudo apt-get update | |
| sudo apt-get install -y --no-install-recommends gobuster ffuf | |
| - name: Install Playwright + chromium (for browser-agent E2E) | |
| run: | | |
| pip install playwright | |
| playwright install --with-deps chromium | |
| - name: Wait for Juice Shop to come up | |
| run: | | |
| for i in {1..60}; do | |
| if curl -fsS http://localhost:3000 > /dev/null 2>&1; then | |
| echo "Juice Shop is ready" | |
| exit 0 | |
| fi | |
| echo "Waiting for Juice Shop ($i/60)..." | |
| sleep 2 | |
| done | |
| echo "Juice Shop failed to start" | |
| exit 1 | |
| - name: Run E2E smoke tests | |
| # The full lifecycle suite (resume/campaign/report) runs on tag | |
| # push as part of release.yml — it's slow (~6 min) and only needs | |
| # to gate releases, not every commit. | |
| env: | |
| PTAI_E2E_JUICESHOP: "1" | |
| PTAI_E2E_JUICESHOP_URL: "http://localhost:3000" | |
| PTAI_E2E_BROWSER: "1" | |
| run: | | |
| pytest tests/test_juiceshop_e2e_smoke.py tests/test_browser_agent_e2e_smoke.py -v --tb=short |