Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 24 additions & 5 deletions .github/workflows/governance-baseline.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,11 +28,30 @@ jobs:
- name: Validate public release boundary
run: bash scripts/validate-public-boundary.sh

- name: Contract tests - storage and collaboration
- name: Contract test - storage contract
run: |
bash tests/test_storage_contract.sh
bash tests/test_collaboration_event_store.sh
bash tests/test_error_exit_codes.sh
set -euo pipefail
bash tests/test_storage_contract.sh 2>&1 | tee /tmp/governance-test-storage.log

- name: Contract test - collaboration event store
run: |
set -euo pipefail
bash tests/test_collaboration_event_store.sh 2>&1 | tee /tmp/governance-test-collaboration.log

- name: Contract test - error exit codes
run: |
set -euo pipefail
bash tests/test_error_exit_codes.sh 2>&1 | tee /tmp/governance-test-exit-codes.log

- name: Public manifest integration tests
run: bash tests/test_public_manifest_pipeline.sh
run: |
set -euo pipefail
bash tests/test_public_manifest_pipeline.sh 2>&1 | tee /tmp/governance-test-manifest.log

- name: Upload governance diagnostics
if: failure()
uses: actions/upload-artifact@v4
with:
name: governance-diagnostics-${{ github.run_id }}
path: /tmp/governance-test-*.log
if-no-files-found: ignore
246 changes: 246 additions & 0 deletions docs/developer/reports/rca-governance-ci-triage-stall-2026-02-28.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,246 @@
# Root Cause Analysis (RCA) Template for DFSS

## Revision history

Date: 2026-02-28
v1.0

---

## **Project Information**

- **Project Name:** task-orchestrator
- **Product/Process Name:** Develop-branch CI governance gate triage workflow
- **Team Members:** Marcus D. (repo owner), Codex agent
- **Date:** 2026-02-28
- **DFSS Phase:** Define / Measure / Analyze / Design / Verify

---

## **1. Define Phase - Problem Understanding**

### **Problem Statement:**

PR #7 (`fix/governance-contract-stability` -> `develop`) remained blocked because required check `governance` repeatedly failed with a generic "exit code 1" in the "Contract tests - storage and collaboration" step. Diagnosis was delayed by an approval-wait stall and repeated inability to retrieve job logs programmatically.

### **Critical to Quality (CTQ) Factors:**

- CI gate determinism (`governance` check must be reproducible)
- Mean time to diagnosis (MTTD) for failed required checks
- Merge throughput on `develop` with strict status checks enabled
- Traceability of test failure causes from CI output

### **Key Customer Requirements (VOC & VOB):**

- **VOC:** Fast, unambiguous feedback for integration PRs; no opaque failures.
- **VOB:** Governance controls must prevent risky merges without creating prolonged, non-actionable stalls.

### **Scope of Analysis:**

- In scope:
- PR #7 state and required checks
- GitHub branch protection/rules behavior on `develop`
- `governance-baseline` workflow step behavior
- Contract-test script reliability and diagnosability
- Agent-based monitoring process used during triage
- Out of scope:
- Runtime behavior of production task orchestration outside CI
- Non-governance workflows unrelated to required checks

### **Preliminary Risk Analysis:**

High risk to delivery flow: required checks block merge, and failure signal is low fidelity. Process risk increased by asynchronous monitoring that can enter approval waits without timely escalation.

---

## **2. Measure Phase - Data Collection & Failure Mode Identification**

### **System Analysis:**

- **System Components & Subsystems:**
- GitHub branch protection on `develop`
- Required checks: `governance`, `boundary-validation`
- GitHub Actions workflow/job execution
- Contract test scripts:
- `tests/test_storage_contract.sh`
- `tests/test_collaboration_event_store.sh`
- `tests/test_error_exit_codes.sh`
- Triage channel: `gh api` log retrieval + agent monitoring

- **Historical Failure Data:**
- PR #7 state: `open`, `mergeable=true`, `mergeable_state=blocked`
- Current head `cbe0600`: `boundary-validation=success`, `governance=failure`
- Failing check annotation: "Process completed with exit code 1." (no script-level root message)
- Repeated log retrieval attempts for job logs returned:
- `error connecting to api.github.com`

- **Process Flow Mapping:**
1. Push commit to PR branch
2. Required checks run on PR
3. `governance` fails at contract tests
4. Annotation exposes only generic exit code
5. Triage attempts log download via `gh api`
6. Log access fails intermittently
7. Monitoring path relies on long waits / approval handling
8. Merge remains blocked

### **Key Measurement Data:**

- Branch protection (`develop`):
- Required checks: `governance`, `boundary-validation` (strict = true)
- Required signatures = true
- Required approving reviews = 0
- Latest evaluated run (head `cbe0600`):
- `governance`: completed/failure
- `boundary-validation`: completed/success
- Both started at 2026-02-28 16:09:01Z; governance failed at 16:09:07Z
- Diagnostic observability:
- Check-run annotation detail level: low (generic exit code only)
- Job log retrieval availability from this environment: unreliable

### **Tools Used:**

- [x] Ishikawa (Fishbone) Diagram
- [ ] Pareto Analysis
- [ ] Process Capability Study
- [ ] Design of Experiments (DOE)
- [x] Historical Data Review

---

## **3. Analyze Phase - Root Cause Identification**

### **Failure Mode Analysis (FMEA Table)**

| Failure Mode | Effect on System | Cause | Severity (1-10) | Occurrence (1-10) | Detection (1-10) | Risk Priority Number (RPN) |
|-------------|----------------|------|----------------|----------------|----------------|----------------------|
| Required check fails with generic output | PR blocked, no immediate remediation path | Step fails with exit 1 but no actionable failure emission | 8 | 6 | 8 | 384 |
| Log retrieval channel unavailable | Triage stalls, long MTTD | Intermittent `gh api` connectivity to GitHub API | 7 | 7 | 9 | 441 |
| Monitoring enters approval-wait state | Extended idle time without progress | Asynchronous wait path without watchdog/escalation timeout | 7 | 5 | 8 | 280 |
| Policy signal confusion (bypass vs required rules) | Decision friction during incident | Branch-protection violations can be bypassed by privileged actor | 6 | 4 | 7 | 168 |

### **Root Cause Determination:**

Primary root causes:

1. **Diagnostic single-point dependency:** CI triage depended on successful remote job-log retrieval; when that channel failed, failure cause could not be determined from check metadata alone.
2. **Low-fidelity failure reporting in governance step:** contract-test stage emitted a generic non-zero exit without durable, script-level diagnostics available in annotations/artifacts.
3. **Monitoring process control gap:** agent-based waiting lacked strict watchdog behavior to force quick fallback from "approval needed" or interrupted states.

Contributing factors:

- Required-signatures and PR-rule bypass messaging introduced perceived ambiguity about the true merge blocker.
- Parallel contract operations increase chance of transient behavior unless test scripts are hardened for CI contention.

### **Key Analysis Techniques Used:**

- [x] 5 Whys Analysis
- [ ] Fault Tree Analysis (FTA)
- [ ] Regression Analysis
- [ ] Statistical Hypothesis Testing

**5 Whys (condensed):**

1. Why blocked? `governance` required check failed.
2. Why unresolved quickly? Failure message was generic (`exit code 1`).
3. Why no deeper detail? Log retrieval from API repeatedly failed in this environment.
4. Why did triage still stall? Monitoring relied on long awaiter path without hard fallback timeout/escalation.
5. Why does this recur risk remain? Governance workflow/scripts do not guarantee machine-readable diagnostics independent of remote log availability.

---

## **4. Design Phase - Corrective Action Implementation**

### **Proposed Design Changes:**

1. Workflow-level diagnostics hardening:
- Wrap each contract test invocation with explicit named output and exit-code capture.
- On failure, print last N lines of per-test logs in-step.
2. Artifact-first failure visibility:
- Upload test logs as artifacts on failure (`if: failure()`).
3. Process watchdog:
- Enforce max wait windows (for monitoring and approvals); auto-fallback to direct polling/local reproduction after timeout.
4. Governance test stability:
- Keep deterministic env setup in contract tests (`TM_AGENT_ID` fixed).
- Retain retry wrappers for high-concurrency collaboration writes to reduce transient CI lock contention.
5. Governance policy clarity:
- Document that `required_approving_review_count=0`; blocker is required checks, not review approval.
- Restrict bypass usage to emergency-only.

### **Mistake-Proofing (Poka-Yoke) Solutions:**

- CI job fails with structured, named error messages by test file.
- Automatic artifact upload prevents total observability loss when console log access is impaired.
- Monitoring runbook enforces timeout + fallback path; no indefinite waiting.

### **Tolerance Design & Robust Engineering:**

- Retry tolerance for transient parallel write failures in collaboration contract test.
- Explicit environment initialization in tests to reduce non-deterministic enforcement side effects.

---

## **5. Verify Phase - Validation & Risk Reduction**

<!-- GUIDANCE:
This phase should only be conducted after the corrective actions proposed in Chapter 4 (Design Phase) have been fully implemented.
The purpose of this section is to validate whether the corrective actions effectively mitigated the issue.
-->

### **Validation Testing:**

Forward-looking validation plan (full implementation still in progress):

1. Run 5 consecutive PR updates through `governance` and track pass/fail + diagnosis time.
2. Inject one controlled failure in contract-test stage and verify:
- named failing script is visible in annotation/log output
- failure logs are downloadable as artifact.
3. Simulate monitoring interruption and confirm watchdog fallback triggers within SLA (<= 5 minutes).

### **Updated Process Controls:**

- Add CI triage SLA:
- If logs unavailable for >5 minutes, switch to fallback path automatically.
- Add operational checklist item:
- Confirm blocker category (`required check` vs `review`) before waiting.
- Add periodic audit:
- Weekly review of branch-protection bypass usage.

### **Revised FMEA & Risk Assessment:**

Target post-mitigation RPNs:

- Generic check failure visibility: 384 -> 120
- Log retrieval unavailability impact: 441 -> 168
- Approval-wait process stall: 280 -> 84
- Policy-signal confusion: 168 -> 72

Residual risk remains **medium** until validation evidence confirms sustained reduction.

### **Final Approval & Sign-Off:**

Pending implementation completion and validation evidence.

**Signatures:**

- **DFSS Black Belt:** Pending
- **Project Manager:** Pending
- **Quality Engineer:** Pending
- **Manufacturing Lead:** N/A (software process)

---

## **Conclusion**

**Final Risk Status:** Not fully mitigated yet. Root causes are identified with actionable corrective design, but verification evidence is still pending.

**Next Steps:**

1. Implement workflow diagnostics + failure artifact publication.
2. Apply and merge governance test hardening changes.
3. Execute verification plan and document measured MTTD improvement.
4. Enforce monitoring watchdog to eliminate approval-wait dead time.

---

**End of Document**
3 changes: 3 additions & 0 deletions src/cli/handlers/admin.py
Original file line number Diff line number Diff line change
Expand Up @@ -423,6 +423,9 @@ def handle_admin(command: str, context: CLIContext) -> Optional[int]:

if command.startswith("phase-"):
pm = tm.phase_manager
if pm is None:
print("Phase management is unavailable in this distribution.")
return 1

if command == "phase-create":
if len(argv) < 3:
Expand Down
19 changes: 16 additions & 3 deletions src/config_manager.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,14 @@
"""

import os
import yaml
import json
from pathlib import Path
from typing import Dict, Any, Optional

try:
import yaml
except ImportError:
yaml = None
from storage_paths import resolve_config_path


Expand Down Expand Up @@ -56,7 +61,11 @@ def _load_config(self) -> Dict[str, Any]:
if self.config_path.exists():
try:
with open(self.config_path, 'r') as f:
config = yaml.safe_load(f) or {}
if yaml:
config = yaml.safe_load(f) or {}
else:
# Fallback keeps runtime dependency-free when PyYAML is absent.
config = json.load(f)
# Merge with defaults to ensure all keys exist
return self._merge_with_defaults(config)
except Exception as e:
Expand Down Expand Up @@ -88,7 +97,11 @@ def _save_config(self, config: Dict = None):

try:
with open(self.config_path, 'w') as f:
yaml.safe_dump(config, f, default_flow_style=False, sort_keys=False)
if yaml:
yaml.safe_dump(config, f, default_flow_style=False, sort_keys=False)
else:
# JSON is valid YAML subset and can be parsed by safe_load later.
json.dump(config, f, indent=2)
except Exception as e:
print(f"Warning: Failed to save config: {e}")

Expand Down
10 changes: 8 additions & 2 deletions src/enforcement.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,8 @@
from datetime import datetime
from enum import Enum

from storage_paths import resolve_db_path

class EnforcementLevel(Enum):
"""Enforcement levels with different behaviors"""
STRICT = "strict" # Block violations
Expand Down Expand Up @@ -148,7 +150,7 @@ def _check_tm_executable(self) -> bool:

def _check_database_initialized(self) -> bool:
"""Check if Task Orchestrator database is initialized"""
db_path = Path.cwd() / ".task-orchestrator" / "tasks.db"
db_path = resolve_db_path()
if not db_path.exists():
return False

Expand Down Expand Up @@ -313,6 +315,10 @@ def enforce_orchestration(self, command: str) -> bool:

elif level == EnforcementLevel.STANDARD:
print(result.guidance)
if (not sys.stdin.isatty()) or os.environ.get("CI") or os.environ.get("GITHUB_ACTIONS"):
print("Non-interactive environment detected; command blocked in STANDARD mode.")
print("Fix orchestration violations or lower enforcement level explicitly.")
return False
response = input("Continue anyway? (y/N): ").lower().strip()
return response in ['y', 'yes']

Expand Down Expand Up @@ -550,4 +556,4 @@ def wrapper(*args, **kwargs):
else:
print(f"Unknown command: {command}")
print("Available commands: status, validate, fix, config")
sys.exit(1)
sys.exit(1)
15 changes: 11 additions & 4 deletions src/tm_production.py
Original file line number Diff line number Diff line change
Expand Up @@ -30,8 +30,14 @@
import hashlib
import time

# Import PhaseManager for phase support
from phase_manager import PhaseManager
# Import PhaseManager for phase support (optional in trimmed/public builds).
try:
from phase_manager import PhaseManager
except ModuleNotFoundError as exc:
if exc.name == "phase_manager":
PhaseManager = None
else:
raise
from storage_paths import resolve_db_path, resolve_storage_root

LOGGER = logging.getLogger("task_orchestrator.tm_production")
Expand Down Expand Up @@ -71,8 +77,9 @@ def __init__(self, agent_id_override=None):
# Multi-Layer Agent ID Resolution System
self.agent_id = self._resolve_agent_id(agent_id_override)

# Initialize PhaseManager for phase support
self.phase_manager = PhaseManager(str(self.db_path))
# Initialize PhaseManager for phase support when available.
# Some public/lean distributions intentionally omit phase_manager.py.
self.phase_manager = PhaseManager(str(self.db_path)) if PhaseManager else None
self.repo_root = self._find_repo_root()

# Initialize error handler
Expand Down
Loading