[CI/CD Assessment] CI/CD Pipelines and Integration Tests Gap Assessment

[github-actions[bot]]

📊 Current CI/CD Pipeline Status

The repository has a mature and well-structured CI/CD setup with 55 standard workflow files and 37 compiled agentic lock workflows. All recent workflow runs show 100% success rates across the 19 distinct workflows captured in recent activity. The pipeline covers build verification, linting, type checking, unit tests with coverage tracking, integration tests, security scanning, and documentation checks — a strong foundation.

---

✅ Existing Quality Gates

The following checks currently run on pull requests:

Workflow	Trigger	What It Checks
Build Verification (build.yml)	PR to main	TypeScript compilation, ESLint, build output, API proxy + CLI proxy unit tests (Node 20 & 22 matrix)
Lint (lint.yml)	PR to main	ESLint on src/, markdownlint on all *.md files
TypeScript Type Check (test-integration.yml)	PR to main	tsc --noEmit strict mode check
Test Coverage (test-coverage.yml)	PR to main	Jest unit tests with coverage, base-vs-PR delta comparison, PR comment with report
CodeQL (codeql.yml)	PR to main	SAST for javascript-typescript and GitHub Actions workflows
Dependency Vulnerability Audit (dependency-audit.yml)	PR to main	npm audit --audit-level=high on main + docs-site packages, SARIF to Security tab
PR Title Check (pr-title.yml)	PR opened/edited	Conventional Commits format (feat, fix, docs, etc.)
Link Check (link-check.yml)	PR touching *.md	Dead link detection via lychee
Chroot Integration Tests (test-chroot.yml)	PR to main	Language runtime support in the chroot container
Integration Tests (test-integration-suite.yml)	PR to main	Broader integration test suite
Smoke Tests	PR to main	Real agent smoke tests: Claude, Codex, Copilot, Copilot BYOK, Gemini, chroot, OTEL tracing, services
Security Guard (agentic)	PR opened/sync	AI-powered security review of diffs
Build & Test (agentic, build-test.md)	PR opened/sync	AI-assisted build and test validation

Test infrastructure: 97 unit test files in src/, 35 integration test files in tests/. Coverage thresholds are set (branches: 30%, functions: 35%, lines/statements: 38%).

---

🔍 Identified Gaps

🔴 High Priority

1. Coverage thresholds are too low
The Jest coverageThreshold in jest.config.js is set to 30% branches, 35% functions, 38% lines/statements. These thresholds are far below industry standards (typically 70-80%) and will not catch meaningful regressions. The test-coverage.yml workflow only blocks merges if coverage decreases from the base branch — it does not enforce a floor.

2. No container image security scanning on PRs
The Squid proxy, agent, and API proxy container images are built from Dockerfiles in containers/. There is no Trivy, Grype, or similar container image vulnerability scan running on PRs. Dependency vulnerabilities inside container layers (Ubuntu base packages, npm packages inside containers) are not caught before merge. This is particularly significant given the security-critical nature of the firewall.

3. No required status checks enforced via branch protection (observable gap)
There is no .github/branch-protection.yml or similar config in the repo. It is not possible to confirm from the repo alone which checks are required before merge. If branch protection is not configured, developers can merge PRs without all checks passing.

4. Performance regression testing is not PR-gated
performance-monitor.yml runs on a schedule (daily) only, not on PRs. Performance regressions in container startup time, proxy latency, or iptables setup can be introduced and merged before being detected.

---

🟡 Medium Priority

5. No mutation testing
Unit tests cover code paths but do not verify test quality. Tools like Stryker can reveal tests that pass even when logic is mutated — a common issue in security-critical code where tests should be verifying behavior, not just coverage.

6. Docker layer and image size tracking
No workflow tracks Docker image size changes between PRs. A PR adding a large dependency to a container image would go unnoticed until it affects pull time in production.

7. Integration test coverage is not reported
tests/ contains 35 integration test files, but there is no coverage report generated from them. Only unit test coverage (97 files in src/) is tracked. Integration tests that exercise Docker, iptables, chroot, and Squid proxy paths are the most critical for this project.

8. No SBOM (Software Bill of Materials) generation
With three container images and multiple npm packages, there is no automated SBOM generation. SBOMs are increasingly required for supply chain security compliance and would complement the existing npm audit checks.

9. Agentic smoke tests are reaction-gated, not automatic on all PRs
smoke-claude, smoke-copilot, smoke-codex, smoke-gemini, and smoke-services run on PR open/sync events but are listed as reaction-triggered (reaction: eyes, heart, etc.). It is unclear if they run automatically for all PRs or only when a reaction is added. If reaction-gated, end-to-end firewall behavior is not validated on every PR.

---

🟢 Low Priority

10. No formatting check (Prettier)
ESLint is configured, but there is no auto-formatter check. Inconsistent formatting can accumulate across PRs, especially as contributors use different editors. Adding prettier --check as a required PR check would eliminate this class of review comment.

11. No changelog/release notes enforcement on PRs
The update-release-notes workflow runs on release, not on PRs. PRs that should be in the changelog (features, bug fixes) are not flagged if they lack the appropriate conventional commit prefix — the PR title check only validates format, not semantic completeness.

12. No spell checking on documentation
link-check.yml validates that links are not broken, and lint:md checks markdown format, but there is no spell checker (e.g., cspell) running on documentation files, which can affect docs quality over time.

13. No test for the awf logs CLI commands
src/commands/logs-stats.ts and src/commands/logs-summary.ts are production commands for GitHub Actions step summary integration. There are no visible integration tests for these commands in the test file listing.

---

📋 Actionable Recommendations

1. Raise coverage thresholds incrementally — High Priority / Low Complexity

Update jest.config.js to raise coverageThreshold to at least 60% across all metrics, with a plan to reach 75% within 6 months. Pair this with the existing test-coverage-improver agentic workflow which already opens PRs to improve coverage.

// jest.config.js — suggested target
coverageThreshold: {
  global: { branches: 60, functions: 65, lines: 68, statements: 68 }
}

2. Add container image scanning to PRs — High Priority / Medium Complexity

Add a container-scan.yml workflow triggered on PRs that change files in containers/**. Use aquasecurity/trivy-action to scan built images and upload SARIF results to the Security tab. This mirrors the existing dependency-audit.yml pattern.

- uses: aquasecurity/trivy-action@master
  with:
    image-ref: 'ghcr.io/github/gh-aw-firewall/agent:latest'
    format: 'sarif'
    output: 'trivy-agent.sarif'
    exit-code: '1'
    severity: 'HIGH,CRITICAL'

3. Gate performance benchmarks on PRs — High Priority / Medium Complexity

Extract the benchmark step from performance-monitor.yml into a separate job that runs on PRs touching src/** or containers/**. Compare against the stored history baseline and fail the PR if a critical metric regresses by more than the defined threshold.

4. Add branch protection with required status checks — High Priority / Low Complexity

Document (or enforce via policy) the minimum set of required checks: Build Verification, Lint, TypeScript Type Check, Test Coverage, CodeQL, Dependency Vulnerability Audit. This prevents merging PRs where checks haven't run or have failed.

5. Add integration test coverage reporting — Medium Priority / Medium Complexity

Instrument npm run test:integration with --coverage and merge the coverage report with unit test coverage. This will reveal which critical integration paths (Docker startup, iptables, chroot) are tested and which are not.

6. Add SBOM generation — Medium Priority / Low Complexity

Add an anchore/sbom-action step to release.yml and/or a PR-triggered workflow for containers/** changes. Attach the SBOM as a release artifact and upload to the Dependency Graph.

7. Add Prettier formatting check — Low Priority / Low Complexity

Run npx prettier --check "src/**/*.ts" as a step in the existing lint.yml workflow. This is a one-line addition.

8. Add cspell spell checking — Low Priority / Low Complexity

Add a cspell.yml or extend lint.yml to run npx cspell "**/*.md" "src/**/*.ts" with a custom dictionary for project-specific terms (awf, squid, iptables, etc.).

---

📈 Metrics Summary

Metric	Value
Total workflow files	55 standard + 37 agentic lock files
Workflows running on PRs	13+ (standard) + 2 agentic (build-test, security-guard)
Recent workflow success rate	100% (19 workflows, most recent 50 runs)
Unit test files	97 (in src/)
Integration test files	35 (in tests/)
Coverage thresholds (current)	Branches: 30%, Functions: 35%, Lines: 38%
SAST tooling	CodeQL (JS/TS + Actions) ✅
Dependency scanning	npm audit → SARIF ✅
Container image scanning	❌ Not present
Performance regression on PRs	❌ Scheduled only
Mutation testing	❌ Not present
Integration test coverage reporting	❌ Not reported

Assessment generated on 2026-06-03. Workflow data based on the most recent 50 GitHub Actions runs.

Generated by CI/CD Pipelines and Integration Tests Gap Assessment · sonnet46 1.3M · ◷

• expires on Jun 10, 2026, 6:22 AM UTC

[github-actions[bot]]

🔮 The ancient spirits stir. The smoke-test agent passed through this discussion, leaving a brief oracle-mark in the scrolls.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• registry.npmjs.org

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "registry.npmjs.org"

See Network Configuration for more information.

🔮 The oracle has spoken through Smoke Codex

[github-actions[bot]]

🔮 The ancient spirits stir, and this smoke test agent was here. May the firewall remain veiled, the winds whitelisted, and the logs speak truth.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• registry.npmjs.org

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "registry.npmjs.org"

See Network Configuration for more information.

🔮 The oracle has spoken through Smoke Codex

[github-actions[bot]]

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• registry.npmjs.org

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "registry.npmjs.org"

See Network Configuration for more information.

🔮 The oracle has spoken through Smoke Codex

[github-actions[bot]]

🔮 The ancient spirits stir, and the smoke test agent has passed through this discussion.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• registry.npmjs.org

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "registry.npmjs.org"

See Network Configuration for more information.

🔮 The oracle has spoken through Smoke Codex

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-06-10T06:22:11.875Z.

Closed by Workflow