Skip to content

Commit 2a23a70

Browse files
quotentirolerquotentiroler
authored
Merge pull request #132 from quotentiroler/dev/testing
🔄 Auto-PR: Merge `dev/testing` → `develop`
2 parents 516e0a5 + 6ab4df0 commit 2a23a70

15 files changed

Lines changed: 1624 additions & 68 deletions

File tree

‎.github/actions/README.md‎

Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,101 @@
1+
# GitHub Actions for SMART-on-FHIR Proxy Testing
2+
3+
This directory contains reusable GitHub Actions for comprehensive testing of the SMART-on-FHIR Proxy application.
4+
5+
## Actions Overview
6+
7+
### 1. `setup-docker-inferno/`
8+
Sets up Docker environment and prepares Inferno ONC Program Edition testing infrastructure.
9+
10+
**Inputs:**
11+
- `test_stage`: Testing stage (alpha, beta, production)
12+
- `fhir_server_url`: FHIR server URL for testing
13+
- `keycloak_url`: Keycloak URL for OAuth testing
14+
- `inferno_version`: Inferno Docker image version (default: latest)
15+
16+
**Outputs:**
17+
- `inferno_container_id`: Container ID of started Inferno instance
18+
- `test_config_path`: Path to generated test configuration
19+
20+
### 2. `run-inferno-tests/`
21+
Executes Inferno ONC Program Edition tests and generates compliance reports.
22+
23+
**Inputs:**
24+
- `test_stage`: Testing stage (alpha, beta, production)
25+
- `inferno_container_id`: Container ID of running Inferno instance
26+
- `test_config_path`: Path to Inferno test configuration
27+
- `fhir_server_url`: FHIR server URL being tested
28+
- `test_timeout`: Maximum test execution time (default: 1800s)
29+
30+
**Outputs:**
31+
- `test_session_id`: ID of created test session
32+
- `test_results_path`: Path to test results directory
33+
- `compliance_status`: Overall compliance status (passed/failed)
34+
35+
### 3. `comprehensive-testing/`
36+
Runs complete test suite: unit tests, integration tests, and ONC Inferno tests.
37+
38+
**Inputs:**
39+
- `test_stage`: Testing stage (alpha, beta, production)
40+
- `deployment_target`: Deployment location (local, fly.io, vps)
41+
- `fhir_server_url`: FHIR server URL for testing
42+
- `keycloak_url`: Keycloak URL for OAuth testing
43+
- `app_version`: Application version being tested
44+
- `skip_unit_tests`: Skip unit tests for deployed environments
45+
- `skip_inferno_tests`: Skip Inferno ONC tests
46+
47+
**Outputs:**
48+
- `unit_test_status`: Unit test results (passed/failed/skipped)
49+
- `integration_test_status`: Integration test results (passed/failed)
50+
- `inferno_test_status`: Inferno test results (passed/failed/skipped)
51+
- `overall_status`: Overall test status (passed/failed)
52+
- `test_reports_path`: Path to all test reports
53+
54+
## Usage Example
55+
56+
```yaml
57+
jobs:
58+
test:
59+
runs-on: ubuntu-latest
60+
steps:
61+
- uses: actions/checkout@v4
62+
63+
- name: Run comprehensive tests
64+
uses: ./.github/actions/comprehensive-testing
65+
with:
66+
test_stage: "beta"
67+
deployment_target: "fly.io"
68+
fhir_server_url: "https://my-app.fly.dev"
69+
keycloak_url: "https://auth.fly.dev"
70+
app_version: "1.2.3"
71+
```
72+
73+
## Test Stages
74+
75+
- **Alpha**: Basic compliance testing with unit and integration tests
76+
- **Beta**: US Core profile testing with extended FHIR compliance
77+
- **Production**: Full ONC certification testing with all required sequences
78+
79+
## Deployment Targets
80+
81+
- **Local**: Tests run in GitHub Actions with local Docker services
82+
- **Fly.io**: Tests run against deployed application on Fly.io
83+
- **VPS**: Tests run against deployed application on your VPS
84+
85+
## Test Reports
86+
87+
All test results are stored in structured directories:
88+
```
89+
testing/
90+
├── alpha/reports/
91+
├── beta/reports/
92+
├── production/reports/
93+
└── summary/
94+
```
95+
96+
Each stage generates comprehensive reports including:
97+
- Unit test coverage
98+
- Integration test results
99+
- FHIR compliance reports
100+
- Inferno ONC test results
101+
- Overall compliance status

‎.github/actions/run-inferno-tests/action.yml‎

Lines changed: 262 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,262 @@
1+
name: 'Run Inferno ONC Tests'
2+
description: 'Executes Inferno ONC Program Edition tests and generates compliance reports'
3+
4+
inputs:
5+
test_stage:
6+
description: 'Testing stage: alpha, beta, or production'
7+
required: true
8+
inferno_container_id:
9+
description: 'Container ID of the running Inferno instance'
10+
required: true
11+
test_config_path:
12+
description: 'Path to the Inferno test configuration file'
13+
required: true
14+
fhir_server_url:
15+
description: 'FHIR server URL being tested'
16+
required: true
17+
test_timeout:
18+
description: 'Maximum time to wait for tests to complete (in seconds)'
19+
required: false
20+
default: '1800'
21+
22+
outputs:
23+
test_session_id:
24+
description: 'ID of the created test session'
25+
value: ${{ steps.run-tests.outputs.session_id }}
26+
test_results_path:
27+
description: 'Path to the test results directory'
28+
value: ${{ steps.run-tests.outputs.results_path }}
29+
compliance_status:
30+
description: 'Overall compliance test status (passed/failed)'
31+
value: ${{ steps.analyze-results.outputs.status }}
32+
33+
runs:
34+
using: 'composite'
35+
steps:
36+
- name: Create Inferno test directories
37+
shell: bash
38+
run: |
39+
echo "📁 Creating Inferno-specific test directories..."
40+
mkdir -p testing/${{ inputs.test_stage }}/reports/inferno
41+
mkdir -p testing/${{ inputs.test_stage }}/data
42+
43+
- name: Verify Inferno is running
44+
shell: bash
45+
run: |
46+
echo "🔍 Verifying Inferno container is running..."
47+
if ! docker ps | grep -q "${{ inputs.inferno_container_id }}"; then
48+
echo "❌ Inferno container is not running"
49+
exit 1
50+
fi
51+
echo "✅ Inferno container is running"
52+
53+
- name: Validate FHIR server for Inferno testing
54+
shell: bash
55+
run: |
56+
echo "🏥 Validating FHIR server for Inferno compliance testing..."
57+
58+
# Test FHIR capability statement (required for Inferno)
59+
echo "Testing FHIR capability statement..."
60+
if curl -f -H "Accept: application/fhir+json" "${{ inputs.fhir_server_url }}/metadata" > testing/${{ inputs.test_stage }}/reports/inferno/capability_statement.json; then
61+
echo "✅ FHIR capability statement retrieved"
62+
else
63+
echo "❌ Failed to retrieve FHIR capability statement"
64+
exit 1
65+
fi
66+
67+
# Test SMART configuration (required for ONC certification)
68+
echo "Testing SMART configuration..."
69+
if curl -f -H "Accept: application/json" "${{ inputs.fhir_server_url }}/.well-known/smart_configuration" > testing/${{ inputs.test_stage }}/reports/inferno/smart_configuration.json; then
70+
echo "✅ SMART configuration retrieved"
71+
else
72+
echo "⚠️ SMART configuration not available (may be expected for some implementations)"
73+
fi
74+
75+
- name: Run Inferno ONC tests
76+
id: run-tests
77+
shell: bash
78+
run: |
79+
echo "🔥 Starting Inferno ONC Program Edition tests for ${{ inputs.test_stage }}..."
80+
81+
# Create test session via Inferno API
82+
echo "Creating new test session..."
83+
SESSION_RESPONSE=$(curl -X POST http://localhost:4567/api/test_sessions \
84+
-H "Content-Type: application/json" \
85+
-d @${{ inputs.test_config_path }} \
86+
--fail --silent --show-error) || {
87+
echo "❌ Failed to create Inferno test session"
88+
echo "Response: $SESSION_RESPONSE"
89+
exit 1
90+
}
91+
92+
# Extract session ID from response
93+
SESSION_ID=$(echo "$SESSION_RESPONSE" | jq -r '.id // empty')
94+
if [ -z "$SESSION_ID" ]; then
95+
echo "❌ Failed to extract session ID from response"
96+
echo "Response: $SESSION_RESPONSE"
97+
exit 1
98+
fi
99+
100+
echo "session_id=$SESSION_ID" >> $GITHUB_OUTPUT
101+
echo "results_path=testing/${{ inputs.test_stage }}/reports/inferno" >> $GITHUB_OUTPUT
102+
echo "✅ Test session created with ID: $SESSION_ID"
103+
104+
# Start the test execution
105+
echo "Starting test execution..."
106+
curl -X POST "http://localhost:4567/api/test_sessions/$SESSION_ID/start" \
107+
-H "Content-Type: application/json" \
108+
--fail || {
109+
echo "❌ Failed to start test execution"
110+
exit 1
111+
}
112+
113+
# Monitor test progress
114+
echo "⏳ Monitoring test progress..."
115+
start_time=$(date +%s)
116+
timeout_time=$((start_time + ${{ inputs.test_timeout }}))
117+
118+
while [ $(date +%s) -lt $timeout_time ]; do
119+
# Check test session status
120+
STATUS_RESPONSE=$(curl -s "http://localhost:4567/api/test_sessions/$SESSION_ID" || echo "{}")
121+
STATUS=$(echo "$STATUS_RESPONSE" | jq -r '.status // "unknown"')
122+
123+
case "$STATUS" in
124+
"completed")
125+
echo "✅ Tests completed successfully"
126+
break
127+
;;
128+
"failed"|"error")
129+
echo "❌ Tests failed or encountered an error"
130+
echo "Status response: $STATUS_RESPONSE"
131+
break
132+
;;
133+
"running"|"started")
134+
echo "🔄 Tests still running... ($(date))"
135+
sleep 30
136+
;;
137+
*)
138+
echo "⚠️ Unknown status: $STATUS"
139+
sleep 30
140+
;;
141+
esac
142+
done
143+
144+
# Final status check
145+
FINAL_STATUS=$(curl -s "http://localhost:4567/api/test_sessions/$SESSION_ID" | jq -r '.status // "timeout"')
146+
if [ "$FINAL_STATUS" != "completed" ]; then
147+
echo "⚠️ Tests did not complete successfully. Final status: $FINAL_STATUS"
148+
fi
149+
150+
echo "🔄 Downloading test results..."
151+
# Download test results
152+
curl -s "http://localhost:4567/api/test_sessions/$SESSION_ID/results" > testing/${{ inputs.test_stage }}/reports/inferno/test_results.json
153+
curl -s "http://localhost:4567/api/test_sessions/$SESSION_ID/report" > testing/${{ inputs.test_stage }}/reports/inferno/test_report.html
154+
155+
- name: Analyze test results
156+
id: analyze-results
157+
shell: bash
158+
run: |
159+
echo "📊 Analyzing Inferno test results..."
160+
161+
RESULTS_FILE="testing/${{ inputs.test_stage }}/reports/inferno/test_results.json"
162+
163+
if [ ! -f "$RESULTS_FILE" ]; then
164+
echo "❌ Test results file not found"
165+
echo "status=failed" >> $GITHUB_OUTPUT
166+
exit 1
167+
fi
168+
169+
# Parse test results
170+
TOTAL_TESTS=$(jq '.test_results | length // 0' "$RESULTS_FILE")
171+
PASSED_TESTS=$(jq '[.test_results[] | select(.result == "pass")] | length // 0' "$RESULTS_FILE")
172+
FAILED_TESTS=$(jq '[.test_results[] | select(.result == "fail")] | length // 0' "$RESULTS_FILE")
173+
SKIPPED_TESTS=$(jq '[.test_results[] | select(.result == "skip")] | length // 0' "$RESULTS_FILE")
174+
175+
echo "📈 Test Results Summary:"
176+
echo " Total: $TOTAL_TESTS"
177+
echo " Passed: $PASSED_TESTS"
178+
echo " Failed: $FAILED_TESTS"
179+
echo " Skipped: $SKIPPED_TESTS"
180+
181+
# Determine overall status
182+
if [ "$FAILED_TESTS" -eq 0 ] && [ "$PASSED_TESTS" -gt 0 ]; then
183+
OVERALL_STATUS="passed"
184+
echo "✅ All tests passed!"
185+
elif [ "$FAILED_TESTS" -gt 0 ]; then
186+
OVERALL_STATUS="failed"
187+
echo "❌ Some tests failed"
188+
else
189+
OVERALL_STATUS="unknown"
190+
echo "⚠️ No clear test results"
191+
fi
192+
193+
echo "status=$OVERALL_STATUS" >> $GITHUB_OUTPUT
194+
195+
- name: Generate compliance report
196+
shell: bash
197+
run: |
198+
echo "📄 Generating compliance report..."
199+
200+
REPORT_FILE="testing/${{ inputs.test_stage }}/reports/inferno/compliance_report.md"
201+
202+
cat > "$REPORT_FILE" << EOF
203+
# ONC Inferno Compliance Report
204+
205+
**Test Stage**: ${{ inputs.test_stage }}
206+
**FHIR Server**: ${{ inputs.fhir_server_url }}
207+
**Test Session ID**: ${{ steps.run-tests.outputs.session_id }}
208+
**Generated**: $(date -u)
209+
**Overall Status**: ${{ steps.analyze-results.outputs.status }}
210+
211+
## Test Configuration
212+
- **Inferno Module**: ONC Program Edition
213+
- **Test Focus**: $(jq -r '.inferno.test_focus // "Unknown"' ${{ inputs.test_config_path }})
214+
- **Test Sequences**: $(jq -r '.inferno.sequences | length // 0' ${{ inputs.test_config_path }}) sequences configured
215+
216+
## Results Summary
217+
- **Test Results**: Available in \`test_results.json\`
218+
- **HTML Report**: Available in \`test_report.html\`
219+
- **FHIR Capability**: Available in \`capability_statement.json\`
220+
- **SMART Config**: Available in \`smart_configuration.json\`
221+
222+
## Next Steps
223+
EOF
224+
225+
if [ "${{ steps.analyze-results.outputs.status }}" = "passed" ]; then
226+
cat >> "$REPORT_FILE" << EOF
227+
✅ **All tests passed** - Ready for ONC certification submission
228+
229+
1. Review detailed test report in \`test_report.html\`
230+
2. Verify all required sequences completed successfully
231+
3. Prepare certification documentation
232+
4. Submit to ONC for official certification review
233+
EOF
234+
else
235+
cat >> "$REPORT_FILE" << EOF
236+
❌ **Some tests failed** - Review and address issues before certification
237+
238+
1. Review failed tests in \`test_results.json\`
239+
2. Check detailed error messages in \`test_report.html\`
240+
3. Address compliance gaps in FHIR implementation
241+
4. Re-run tests after fixes are implemented
242+
EOF
243+
fi
244+
245+
echo "✅ Compliance report generated at $REPORT_FILE"
246+
247+
- name: Export container logs
248+
if: always()
249+
shell: bash
250+
run: |
251+
echo "📋 Exporting Inferno container logs..."
252+
docker logs "${{ inputs.inferno_container_id }}" > testing/${{ inputs.test_stage }}/reports/inferno/inferno_container.log 2>&1 || true
253+
echo "✅ Container logs exported"
254+
255+
- name: Cleanup Inferno container
256+
if: always()
257+
shell: bash
258+
run: |
259+
echo "🧹 Cleaning up Inferno container..."
260+
docker stop "${{ inputs.inferno_container_id }}" || true
261+
docker rm "${{ inputs.inferno_container_id }}" || true
262+
echo "✅ Inferno container cleaned up"

0 commit comments

Comments
 (0)