Skip to content

HARNESS.md

Latest commit

 

History

History
622 lines (475 loc) · 13.4 KB

HARNESS.md

File metadata and controls

622 lines (475 loc) · 13.4 KB
Raw

ARAL Conformance Test Harness

Version: 1.0
Status: Release Candidate
Date: 2026-01-14

1. Overview

This document describes the ARAL Conformance Test Harness, which validates that implementations correctly adhere to the ARAL specification requirements.

1.1 Purpose

The test harness enables:

  • Validation of ARAL conformance profiles (CORE, ORCH, INTEROP)
  • Certification of agent implementations
  • Regression testing during development
  • Interoperability testing between implementations

1.2 Test Categories

Category Tests Coverage
Layer Tests 161 All normative requirements
Security Tests 60 ARAL-SECURITY requirements
Protocol Tests 26 L7 protocol validation
Schema Tests 8 JSON schema validation
Integration Tests 15 End-to-end scenarios

2. Installation

2.1 NPM Package

npm install -g @aral-standard/conformance

2.2 From Source

git clone https://github.com/personamanagmentlayer/aral-standard.git
cd aral-standard/tests/harness
npm install
npm link

2.3 Docker

docker pull aral/conformance-harness:1.0
docker run -v $(pwd):/workspace aral/conformance-harness:1.0 test --profile ARAL-CORE

3. Running Tests

3.1 Basic Usage

# Test a specific profile
aral-conformance test --profile ARAL-CORE ./my-agent

# Test all profiles
aral-conformance test --profile ALL ./my-agent

# Run specific test suite
aral-conformance test --suite security ./my-agent

# Verbose output
aral-conformance test --profile ARAL-CORE --verbose ./my-agent

3.2 Configuration File

Create aral-conformance.json in your project:

{
  "version": "1.0",
  "agent": {
    "name": "my-agent",
    "endpoint": "http://localhost:8080",
    "profiles": ["ARAL-CORE"],
    "timeout_ms": 30000
  },
  "tests": {
    "include": ["L1-*", "L2-*", "L3-*", "L4-*", "L5-*"],
    "exclude": ["ARAL-L3-011"],
    "parallel": true,
    "retry": 3
  },
  "reporting": {
    "format": "json",
    "output": "./conformance-report.json"
  }
}

Run with config:

aral-conformance test --config aral-conformance.json

4. Test Structure

4.1 Test Format

Each test follows this structure:

{
  "id": "ARAL-L1-001",
  "requirement": "Runtime MUST provide unique agent instance ID",
  "profile": "ARAL-CORE",
  "level": "MUST",
  "test": {
    "type": "runtime",
    "steps": [
      {
        "action": "start_agent",
        "expect": "agent_running"
      },
      {
        "action": "get_instance_id",
        "validate": "uuid_v4",
        "expect": "unique_id"
      }
    ]
  }
}

4.2 Test Types

Type Description Example
runtime Tests L1 runtime behavior Health checks, lifecycle
memory Tests L2 memory operations State persistence, TTL
capability Tests L3 capability invocation Input validation, timeouts
reasoning Tests L4 decision making Planning, tool selection
persona Tests L5 persona constraints Permission enforcement
orchestration Tests L6 multi-agent coordination Circuit breakers, routing
protocol Tests L7 external protocols MCP, REST, gRPC
security Tests security requirements Auth, encryption, audit
schema Tests JSON schema validation Envelope, capabilities

5. Layer-Specific Tests

5.1 Layer 1: Runtime (ARAL-L1-001 to ARAL-L1-010)

# Test agent lifecycle
aral-conformance test --suite L1 --test lifecycle

# Test resource limits
aral-conformance test --suite L1 --test resources

# Test health endpoints
aral-conformance test --suite L1 --test health

Tests:

  • ✅ Unique instance ID generation
  • ✅ Graceful shutdown with timeout
  • ✅ Health check endpoint
  • ✅ Resource quota enforcement
  • ✅ Fallback on resource exhaustion
  • ✅ Lifecycle event logging
  • ✅ Configuration hot reload
  • ✅ Maximum execution time
  • ✅ Metrics endpoint (Prometheus)
  • ✅ Backpressure mechanisms

5.2 Layer 2: Memory (ARAL-L2-001 to ARAL-L2-010)

aral-conformance test --suite L2

Tests:

  • ✅ Working memory implementation
  • ✅ TTL-based expiration
  • ✅ Vector similarity search
  • ✅ Atomic read-modify-write
  • ✅ No plaintext secrets
  • ✅ Memory partitioning
  • ✅ Audit logging of writes
  • ✅ Size limits with eviction
  • ✅ Snapshot support
  • ✅ Concurrent access safety

5.3 Layer 3: Capabilities (ARAL-L3-001 to ARAL-L3-012)

aral-conformance test --suite L3

Tests:

  • ✅ Input/output schema declaration
  • ✅ Permission declaration
  • ✅ Timeout handling
  • ✅ Retry with backoff
  • ✅ Input validation
  • ✅ Output validation
  • ✅ Invocation logging with trace ID
  • ✅ Rate limiting
  • ✅ Error handling
  • ✅ Permission enforcement
  • ✅ Cost/latency estimation
  • ✅ Cancellation support

5.4 Layer 4: Reasoning (ARAL-L4-001 to ARAL-L4-012)

aral-conformance test --suite L4

Tests:

  • ✅ Decision logging
  • ✅ Reasoning chain traceability
  • ✅ Constraint compliance
  • ✅ Tool selection logic
  • ✅ Planning capabilities
  • ✅ Context management
  • ✅ Fallback strategies
  • ✅ Deterministic mode support
  • ✅ Max reasoning depth
  • ✅ Timeout handling
  • ✅ Error recovery
  • ✅ Audit trail generation

5.5 Layer 5: Persona (ARAL-L5-001 to ARAL-L5-010)

aral-conformance test --suite L5

Tests:

  • ✅ Persona schema validation
  • ✅ Immutability at runtime
  • ✅ Constraint enforcement
  • ✅ Permission boundaries
  • ✅ Identity verification
  • ✅ Signature validation
  • ✅ Version compatibility
  • ✅ Override prevention
  • ✅ Startup validation
  • ✅ Contract adherence

5.6 Layer 6: Orchestration (ARAL-L6-001 to ARAL-L6-013)

aral-conformance test --suite L6 --profile ARAL-ORCH

Tests:

  • ✅ Agent routing
  • ✅ Persona constraint enforcement
  • ✅ Circuit breaker pattern
  • ✅ Load balancing
  • ✅ Failure handling
  • ✅ Routing decision logging
  • ✅ Priority queues
  • ✅ Request timeout
  • ✅ Trace context propagation
  • ✅ Bulkhead pattern
  • ✅ Graceful degradation
  • ✅ Agent discovery
  • ✅ Inter-agent authorization

5.7 Layer 7: Protocol (ARAL-L7-001 to ARAL-L7-013)

aral-conformance test --suite L7 --profile ARAL-INTEROP

Tests:

  • ✅ MCP compatibility
  • ✅ Message validation
  • ✅ Envelope transformation
  • ✅ Authentication
  • ✅ Rate limiting
  • ✅ Request signing
  • ✅ External interaction logging
  • ✅ Protocol error handling
  • ✅ Protocol negotiation
  • ✅ TLS enforcement
  • ✅ Compression support
  • ✅ Request timeout
  • ✅ OpenAPI/AsyncAPI spec

6. Security Tests (ARAL-S-001 to ARAL-S-060)

aral-conformance test --suite security --all

6.1 Authentication & Authorization

aral-conformance test --suite security --test auth

Tests:

  • OAuth 2.0 / OIDC compliance
  • mTLS support
  • Capability-based authorization
  • Least privilege enforcement
  • Token lifetime boundaries
  • Failed auth attempt logging
  • Rate limiting on auth
  • Service-to-service auth (SPIFFE)

6.2 Data Protection

aral-conformance test --suite security --test data-protection

Tests:

  • TLS 1.3+ in transit
  • Encryption at rest
  • PII protection
  • Secret redaction in logs
  • Data retention policies
  • Verifiable deletion
  • Key rotation
  • KMS/HSM integration

6.3 Threat Validation

aral-conformance test --suite security --test threats

Tests STRIDE vectors for each layer:

  • Spoofing resistance
  • Tampering detection
  • Repudiation prevention
  • Information disclosure prevention
  • Denial of service mitigation
  • Elevation of privilege prevention

7. Report Generation

7.1 Output Formats

# JSON report
aral-conformance test --profile ARAL-CORE --format json > report.json

# HTML report
aral-conformance test --profile ARAL-CORE --format html > report.html

# Markdown report
aral-conformance test --profile ARAL-CORE --format markdown > report.md

# JUnit XML (for CI)
aral-conformance test --profile ARAL-CORE --format junit > junit.xml

7.2 Report Structure

{
  "conformance_report": {
    "version": "1.0",
    "timestamp": "2026-01-14T10:00:00Z",
    "agent": {
      "name": "my-agent",
      "version": "1.0.0",
      "profiles": ["ARAL-CORE"]
    },
    "results": {
      "total": 68,
      "passed": 65,
      "failed": 2,
      "skipped": 1,
      "success_rate": 95.6
    },
    "tests": [
      {
        "id": "ARAL-L1-001",
        "status": "passed",
        "duration_ms": 150
      },
      {
        "id": "ARAL-L2-003",
        "status": "failed",
        "error": "Vector search not implemented",
        "duration_ms": 200
      }
    ],
    "certification": {
      "eligible": false,
      "reason": "2 MUST requirements failed"
    }
  }
}

8. CI/CD Integration

8.1 GitHub Actions

name: ARAL Conformance

on: [push, pull_request]

jobs:
  conformance:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Install ARAL Test Harness
        run: npm install -g @aral-standard/conformance

      - name: Start Agent
        run: ./start-agent.sh

      - name: Run Conformance Tests
        run: aral-conformance test --profile ARAL-CORE --format junit > junit.xml

      - name: Publish Test Results
        uses: EnricoMi/publish-unit-test-result-action@v2
        if: always()
        with:
          files: junit.xml

8.2 GitLab CI

conformance:
  stage: test
  image: aral/conformance-harness:1.0
  script:
    - ./start-agent.sh
    - aral-conformance test --profile ARAL-CORE --format junit
  artifacts:
    reports:
      junit: junit.xml

9. Certification

9.1 Self-Certification

To self-certify ARAL conformance:

  1. Run full test suite for your profile
  2. Generate conformance report
  3. Publish report publicly (GitHub, website)
  4. Add conformance badge to README
# Run tests
aral-conformance test --profile ARAL-CORE --format json > conformance-report.json

# Validate report
aral-conformance validate-report conformance-report.json

# Generate badge
aral-conformance badge --profile ARAL-CORE --output badge.svg

9.2 Badge Usage

[![ARAL-CORE Conformant](https://img.shields.io/badge/ARAL--CORE-v1.0-green.svg)](https://github.com/your-org/your-agent/blob/main/conformance-report.json)

9.3 Third-Party Certification

For official ARAL certification:

  1. Submit implementation to certification@aral-standard.org
  2. Provide conformance report
  3. Pass third-party audit
  4. Receive certification certificate (valid 12 months)
  5. Annual recertification required

10. Test Vectors

Pre-defined test vectors are available in tests/vectors/:

File Purpose
valid-envelope.json Valid envelope format
invalid-envelope-no-trace.json Missing trace_id
valid-task.json Valid L4 task
invalid-task-priority.json Invalid priority value
tool-call-with-secret.json Secret handling test
replay-message.json Replay attack test
sample-conformance-report.json Example report

11. Troubleshooting

11.1 Common Issues

Agent not responding:

# Check agent health
curl http://localhost:8080/health

# Increase timeout
aral-conformance test --profile ARAL-CORE --timeout 60000

Schema validation failures:

# Validate schemas independently
aral-conformance validate-schema --schema envelope ./my-envelope.json

Test failures on SHOULD requirements:

# Run only MUST requirements
aral-conformance test --profile ARAL-CORE --level MUST

11.2 Debug Mode

# Enable debug logging
ARAL_DEBUG=1 aral-conformance test --profile ARAL-CORE --verbose

# Trace HTTP requests
ARAL_TRACE_HTTP=1 aral-conformance test --profile ARAL-CORE

12. Contributing Tests

To add new tests:

  1. Create test definition in tests/suites/
  2. Follow test format schema
  3. Add test vectors if needed
  4. Update this documentation
  5. Submit PR with test rationale

Example test contribution:

{
  "id": "ARAL-L1-011",
  "requirement": "Runtime SHOULD support custom metrics",
  "profile": "ARAL-CORE",
  "level": "SHOULD",
  "test": {
    "type": "runtime",
    "steps": [
      {
        "action": "register_custom_metric",
        "params": { "name": "test.counter", "type": "counter" },
        "expect": "success"
      },
      {
        "action": "increment_metric",
        "params": { "name": "test.counter" },
        "expect": "metric_incremented"
      }
    ]
  }
}

13. Resources

14. License

ARAL Conformance Test Harness is licensed under Apache 2.0.

See LICENSE for full details.