Skip to content
CI

Merge pull request #668 from dangerous-funds/feature/issue-477-diagno… #501

Sign in to view logs

Merge pull request #668 from dangerous-funds/feature/issue-477-diagno…

Merge pull request #668 from dangerous-funds/feature/issue-477-diagno… #501

Workflow file for this run

.github/workflows/ci.yml at a14481a
name: CI
on:
push:
branches: [main]
pull_request:
jobs:
test:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ["3.11", "3.12"]
steps:
- uses: actions/checkout@v4
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
cache: pip
- name: Install dependencies
run: |
pip install -r requirements.txt
pip install ruff black nbconvert nbformat jupyter-client ipykernel
- name: Lint
run: |
ruff check .
black --check .
- name: Source package integrity check (Issue #540)
run: python scripts/check_package_integrity.py
- name: Environment contract docs drift check (Issue #544)
run: python scripts/generate_env_contract_docs.py --check
- name: Dead-path detection report (Issue #547)
# Informational — surfaces retirement candidates without failing the
# build. Use `make dead-path-report` locally, or pass --strict to
# gate CI once the initial candidate backlog above has been triaged.
run: python scripts/detect_dead_paths.py
- name: Check verify_chain called after every joblib.load in detection/
run: |
python - <<'EOF'
import re, sys, pathlib
issues = []
for f in pathlib.Path("detection").rglob("*.py"):
lines = f.read_text().splitlines()
for i, line in enumerate(lines):
if re.search(r"joblib\.load\(", line):
# Check the next 5 lines for verify_chain
window = "\n".join(lines[i : i + 6])
if "verify_chain" not in window:
issues.append(f"{f}:{i+1}: joblib.load without nearby verify_chain call")
if issues:
print("verify_chain enforcement failures:")
for issue in issues:
print(" ", issue)
sys.exit(1)
else:
print("verify_chain check passed.")
EOF
- name: Validate Grafana dashboard JSON schema (Issue #242)
run: |
python - <<'EOF'
import json, pathlib, sys
REQUIRED_KEYS = {"schemaVersion", "title", "panels"}
errors = []
for f in pathlib.Path("monitoring/grafana/dashboards").glob("*.json"):
try:
data = json.loads(f.read_text())
except json.JSONDecodeError as e:
errors.append(f"{f}: invalid JSON — {e}")
continue
missing = REQUIRED_KEYS - data.keys()
if missing:
errors.append(f"{f}: missing required keys {missing}")
if not isinstance(data.get("panels"), list):
errors.append(f"{f}: 'panels' must be a list")
if errors:
print("Grafana dashboard validation failures:")
for e in errors:
print(" ", e)
sys.exit(1)
else:
print(f"All Grafana dashboards valid ({len(list(pathlib.Path('monitoring/grafana/dashboards').glob('*.json')))} files).")
EOF
- name: Validate markdown links (Issue #263)
run: |
npm install -g markdown-link-check
markdown-link-check docs/security_threat_model.md
markdown-link-check docs/security.md
markdown-link-check CONTRIBUTING.md
- name: Check import cycles (Issue #546)
run: |
python scripts/check_import_cycles.py
# Exit code 2 = cycles found → fail CI.
# Exit code 1 = fatal error (unreadable file, bad args) → also fail.
- name: Probe optional dependencies (Issue #542)
run: |
python -m utils.dependency_probe --groups ml_core cryptography prometheus
# Probes core required groups; missing optional deps are warnings only.
# Add --require <group> here if a group becomes mandatory.
- name: Validate README bash examples (Issue #548)
run: |
python scripts/validate_readme_examples.py --docs README.md docs/
# Exits 2 if any python -m or python <file> reference is broken.
# make <target> mismatches are always warnings (never block CI).
- name: Validate notebooks — structure (Issue #549)
run: |
python scripts/validate_notebooks.py
# Checks JSON validity, nbformat ≥ 4, kernelspec, empty cells.
# Does NOT enforce output clearing here — that's the --check-outputs flag.
- name: Validate notebooks — outputs cleared (Issue #549)
run: |
python scripts/validate_notebooks.py --check-outputs
# Notebooks must not commit cell outputs (diff bloat / data leakage).
# Add metadata.keep_outputs=true in the notebook to opt out.
- name: Validate environment configuration contracts (config/contracts.py)
# Exercises every runtime mode's contract (config/contracts.py) end-to-end
# against a fully-populated environment, the same way `make check-env`
# does for a developer. Catches contract/entry-point drift — e.g. a
# renamed Config attribute a check still references — that unit tests
# mocking individual attributes could miss.
run: |
openssl genpkey -algorithm ed25519 -out /tmp/ci_signing_key.pem
openssl pkey -in /tmp/ci_signing_key.pem -pubout -out /tmp/ci_jwt_public_key.pem
MODEL_DIR=./models \
RISK_SCORE_DB_URL=sqlite:///:memory: \
WATCHED_ASSET_PAIRS=USDC:GA5ZSEJYBY3RJRC5AVCIA5MOP4RHTM335X2KGX3IHOJAPP5RE34K4KZVN \
API_KEYS=ci-placeholder-hash \
LEDGERLENS_CONTRACT_ID=ci-placeholder-contract \
LEDGERLENS_SUBMITTER_SECRET=ci-placeholder-secret \
JWT_PUBLIC_KEY_PATH=/tmp/ci_jwt_public_key.pem \
python -m scripts.check_env --all
- name: Execute Benford explainer notebook
run: |
cd notebooks
jupyter nbconvert --to notebook --execute \
--ExecutePreprocessor.timeout=120 \
--inplace benford_explainer.ipynb
- name: Check typed service boundaries (utils/boundaries.py)
run: python scripts/check_service_boundaries.py
- name: Check module dependency rules (config/module_boundaries.yml)
run: python scripts/check_module_dependencies.py
- name: Check public API compatibility (tests/fixtures/api_baseline.json)
run: python scripts/check_api_compatibility.py
- name: Check CLI command contracts (scripts/cli_contracts.py)
run: python scripts/check_cli_contracts.py
- name: Test
run: pytest -q --junitxml=reports/ci/junit_results.xml
continue-on-error: true
- name: Triage build failures (Issue #539)
if: always()
run: |
mkdir -p reports/ci
if [ -f reports/ci/junit_results.xml ]; then
python -m scripts.triage_build_failures \
--input reports/ci/junit_results.xml \
--format junit-xml \
--output-dir reports/triage \
--baseline reports/triage/triage_baseline.json \
--compare || true
fi
- name: Feature compatibility check (Issue #532)
if: always()
run: |
python -m scripts.check_feature_compat \
--model-dir models/ \
--no-report || true
schema-compatibility:
name: Avro Schema Compatibility
runs-on: ubuntu-latest
# A separate job rather than a step in `test`: this needs fetch-depth: 0 to
# read the baseline from git, and the test matrix runs checkout twice
# (3.11 and 3.12). Running it once here avoids doubling the clone cost.
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Set up Python 3.11
uses: actions/setup-python@v5
with:
python-version: "3.11"
cache: pip
- name: Install dependencies
run: pip install -r requirements.txt
- name: Fetch the base branch
# On a pull_request the baseline is the target branch; on a push to
# main it is main's previous state, which the full clone already has.
env:
BASE_REF: ${{ github.event.pull_request.base.ref || github.ref_name }}
run: git fetch --no-tags origin "$BASE_REF"
- name: Check Avro schema compatibility
run: make check-schema-compatibility
# -----------------------------------------------------------------------
# Issue #545 — Static analysis gates (mypy + bandit + radon)
# -----------------------------------------------------------------------
- name: Install static analysis tools (issue #545)
run: pip install bandit>=1.7.9 radon>=6.0.1
- name: Static analysis gate — mypy type checking
run: |
python -m mypy \
--config-file pyproject.toml \
--no-error-summary \
detection ingestion streaming ci_metrics benchmarks utils config.py \
|| true
# Non-blocking on first introduction — existing code is not mypy-strict.
# Tracked as follow-up: tighten to exit 1 once all modules are annotated.
- name: Static analysis gate — bandit security linting
run: |
python -m bandit \
-r detection ingestion streaming ci_metrics benchmarks \
--severity-level high \
--confidence-level medium \
--quiet \
|| true
# Non-blocking on first introduction.
- name: Static analysis gate — radon cyclomatic complexity
run: |
python scripts/static_analysis_gate.py \
--skip-mypy \
--skip-bandit \
--complexity-max 10 \
--targets detection ingestion streaming ci_metrics benchmarks
# -----------------------------------------------------------------------
# Issue #541 — Dependency lockfile verification
# -----------------------------------------------------------------------
- name: Verify dependency lockfile (issue #541)
run: |
python scripts/verify_lockfile.py --check-unpinned
# -----------------------------------------------------------------------
# Issue #537 — Benchmark dataset integrity check
# -----------------------------------------------------------------------
- name: Validate benchmark datasets (issue #537)
run: |
python - <<'EOF'
import json, sys
from benchmarks.datasets import BenchmarkRegistry
registry = BenchmarkRegistry()
manifest = registry.manifest()
errors = []
for entry in manifest:
name = entry["name"]
ds = registry.get(name)
if not len(ds.trades) > 0:
errors.append(f"{name}: zero trades")
if ds.labels.dtype != bool:
errors.append(f"{name}: labels dtype is {ds.labels.dtype}")
# Verify checksum is stable
c1 = registry.checksum(name)
c2 = registry.checksum(name)
if c1 != c2:
errors.append(f"{name}: checksum not deterministic")
if errors:
print("Benchmark dataset validation failures:")
for e in errors:
print(" ", e)
sys.exit(1)
print(json.dumps(manifest, indent=2, default=str))
print(f"\n{len(manifest)} benchmark datasets validated.", file=sys.stderr)
EOF
# -----------------------------------------------------------------------
# Issue #538 — CI metrics regression monitoring self-test
# -----------------------------------------------------------------------
- name: CI metrics regression monitoring self-test (issue #538)
run: |
python - <<'EOF'
from ci_metrics import CIRunRecord, MetricSnapshot, record_run
from ci_metrics.store import MetricsStore
from pathlib import Path
import tempfile, os
with tempfile.TemporaryDirectory() as tmp:
store_path = Path(tmp) / "test_history.jsonl"
# Simulate 5 runs with stable metrics
for i in range(5):
record = CIRunRecord(
run_id=f"ci-selftest-{i}",
commit_sha="abc123",
branch="ci-selftest",
timestamp_utc="2024-01-01T00:00:00Z",
metrics=[
MetricSnapshot(name="test_pass_rate", value=1.0),
MetricSnapshot(name="runtime_s", value=10.0, higher_is_better=False),
],
)
record_run(record, store_path=store_path)
store = MetricsStore(store_path)
assert len(store) == 5, f"Expected 5 records, got {len(store)}"
# Verify no regression on stable series
from ci_metrics.regression import RegressionDetector
detector = RegressionDetector(store)
latest = CIRunRecord(
run_id="ci-selftest-latest",
commit_sha="def456",
branch="ci-selftest",
timestamp_utc="2024-01-02T00:00:00Z",
metrics=[
MetricSnapshot(name="test_pass_rate", value=1.0),
],
)
store.append(latest)
alerts = detector.check(latest)
assert not any(a.severity == "critical" for a in alerts), \
f"Unexpected critical alert on stable metrics: {alerts}"
print("CI metrics self-test passed.")
EOF
mutation-test:
name: Mutation Testing (≥80% score)
runs-on: ubuntu-latest
# Run mutation testing only on Python 3.11 to keep CI time under 15 minutes.
# It runs in parallel with the test matrix so it does not block PR feedback.
steps:
- uses: actions/checkout@v4
- name: Set up Python 3.11
uses: actions/setup-python@v5
with:
python-version: "3.11"
cache: pip
- name: Install dependencies
run: pip install -r requirements.txt
- name: Run mutation tests on core scoring path
# --paths-to-mutate limits scope to the three core modules so the job
# completes in < 15 minutes. Mutmut restores every mutant after testing,
# so no mutated code is ever persisted to disk or the workspace.
run: |
mutmut run \
--paths-to-mutate "detection/benford_engine.py,detection/feature_engineering.py,detection/model_inference.py" \
--runner "python -m pytest -x -q --timeout=30 -m 'not integration and not slow' \
tests/test_benford.py \
tests/test_benford_ci.py \
tests/test_feature_engineering.py \
tests/test_model_inference.py" \
--no-progress || true
- name: Print mutation results summary
run: mutmut results || true
- name: Enforce ≥80% mutation score threshold
# Exits 1 (failing the CI step) if fewer than 80% of mutations are killed.
# Prints surviving mutations so they can be opened as follow-up issues.
run: python scripts/check_mutation_score.py --threshold 80