Skip to content

testing-strategy.md

Latest commit

 

History

History
162 lines (113 loc) · 4.27 KB

testing-strategy.md

File metadata and controls

162 lines (113 loc) · 4.27 KB

Testing Strategy

Test organization and execution across the two-layer architecture

Quick Reference: Two-Layer Architecture | Testing Guide | Workflow

This document describes how tests are structured and run across both layers of the architecture.

Infrastructure Tests (tests/infra_tests/)

  • Verify build orchestration works
  • Test validation logic
  • Check file integrity checking
  • Validate PDF generation
  • No dependency on scientific code

Command:

uv run pytest tests/infra_tests/ --cov=infrastructure

Project Tests (projects/{name}/tests/)

  • Test algorithm correctness
  • Verify statistical computations
  • Check data processing
  • Validate visualization output
  • No dependency on build infrastructure

Command:

uv run pytest projects/{name}/tests/ --cov=projects/{name}/src

Integration Tests (tests/integration/)

  • End-to-end pipeline validation
  • Script execution testing
  • Layer interaction verification
  • Output completeness checking

Command:

uv run pytest tests/integration/ --cov=projects/{name}/src --cov=infrastructure

Full Test Suite

# Local all-project union gate (75%); CI project jobs enforce each exemplar's own 90% floor
uv run pytest tests/ projects/{name}/tests/ --cov=infrastructure --cov=projects/{name}/src --cov-fail-under=75

# Generate HTML coverage report
uv run pytest tests/ projects/{name}/tests/ --cov=infrastructure --cov=projects/{name}/src --cov-report=html
open htmlcov/index.html

Coverage Requirements

Layer Minimum Current Target
Infrastructure (infrastructure/) 60% live → ../_generated/COUNTS.md gated by CI; never hardcode the percentage in prose
Project (projects/{name}/src/) 90% live → COUNTS.md per-exemplar live percentages live there

Best Practices

For Infrastructure Development

✅ Do:

  • Write generic, reusable code
  • Document with project-independent examples
  • Test extensively with real scenarios
  • Handle errors gracefully
  • Provide clear logging

❌ Don't:

  • Import scientific modules
  • Assume specific research domain
  • Skip tests to ship features
  • Hardcode project-specific values
  • Mix concerns (building vs. computation)

For Scientific Development

✅ Do:

  • Use infrastructure tools for document management
  • Follow thin orchestrator pattern in projects/{name}/scripts/
  • Implement algorithms in projects/{name}/src/ modules
  • Test with data
  • Document domain-specific concepts

❌ Don't:

  • Duplicate build/validation logic
  • Implement document generation in scripts
  • Skip layer abstraction
  • Mix orchestration with computation

Logging Best Practices

# In project scripts — mark layer transitions
import logging
logger = logging.getLogger(__name__)

logger.info("[LAYER-2-PROJECT] Starting simulation...")
logger.info("[LAYER-1-INFRASTRUCTURE] Using FigureManager for output...")
# In build scripts — mark phase transitions
log_info "━━━ LAYER 1: Infrastructure Validation ━━━"
log_info "━━━ LAYER 2: Scientific Computation ━━━"

Troubleshooting

Import Errors

Error: ModuleNotFoundError: No module named 'project.src'

Solution: Ensure tests/conftest.py includes projects/{name}/ on path:

import sys
sys.path.insert(0, os.path.join(repo_root, "projects", project_name))

Layer Violations

Error: Infrastructure module imports from project

Solution: Refactor to remove dependency or move code to appropriate layer

Check:

# Find infrastructure imports of project code
grep -r "from projects\." infrastructure/
grep -r "import projects\." infrastructure/

Mixed Concerns

Error: Build logic in project module

Solution: Move to infrastructure layer or extract into separate module

Related Documentation: