Nightly Soak Test #119
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| name: Nightly Soak Test | |
| # Long-duration sustained-load test. Catches the class of failures that | |
| # only surface under hours-scale traffic: slow memory leaks, connection- | |
| # pool exhaustion, Redis key-bloat, p99 latency degradation, orphaned | |
| # TTL sorted-set entries. | |
| # | |
| # Test: SoakIntegrationTest (@Tag("soak")) | |
| # Invariants asserted after 10 minutes at ~100 ops/s: | |
| # S1. JVM heap end/start ratio < 2.0 (no runaway leak) | |
| # S2. Avg latency final-minute vs baseline-minute < 3x (no degradation) | |
| # S3. reservation:res_* keys bounded by ops × 1.1; idem:* by ops × 2.1 | |
| # S4. No orphaned reservation:ttl zset entries (every member has a hash) | |
| # | |
| # Excluded from PR CI — the soak profile has surefire group=soak and the | |
| # default build excludes that group. Run locally with: | |
| # mvn test -Psoak --file cycles-protocol-service/pom.xml -pl cycles-protocol-service-api | |
| # | |
| # Parameters are tuned for ubuntu-latest (2 vCPU, 7 GB RAM). A longer | |
| # soak on a proper perf box can be triggered manually via | |
| # workflow_dispatch with a larger soak.duration.minutes. | |
| on: | |
| schedule: | |
| - cron: '30 6 * * *' # 06:30 UTC daily (runs after the property-tests workflow) | |
| workflow_dispatch: | |
| inputs: | |
| duration_minutes: | |
| description: 'Soak duration in minutes (default 10)' | |
| required: false | |
| default: '10' | |
| target_rps: | |
| description: 'Target throughput req/s (default 100)' | |
| required: false | |
| default: '100' | |
| permissions: read-all | |
| jobs: | |
| soak-test: | |
| name: Sustained-load stability | |
| runs-on: ubuntu-latest | |
| # Timeout = soak duration + generous overhead (Docker pull, Maven warmup, | |
| # JVM startup, post-run assertions). 90 min covers default 10-min run, | |
| # 30-min workflow_dispatch runs, and up to a ~60-min deep soak. For | |
| # longer-duration dispatches, the soak profile's forkedProcessTimeout | |
| # (set to 4500s in the pom) is the binding constraint; bump both if you | |
| # want >60-min runs. | |
| timeout-minutes: 90 | |
| permissions: | |
| contents: read | |
| steps: | |
| - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | |
| - name: Set up JDK 21 | |
| uses: actions/setup-java@03ad4de0992f5dab5e18fcb136590ce7c4a0ac95 # v5 | |
| with: | |
| distribution: temurin | |
| java-version: 21 | |
| cache: maven | |
| - name: Run soak test | |
| run: > | |
| mvn -B test | |
| --file cycles-protocol-service/pom.xml | |
| -pl cycles-protocol-service-api -am | |
| -Psoak | |
| -Dsoak.duration.minutes=${{ github.event.inputs.duration_minutes || '10' }} | |
| -Dsoak.target.rps=${{ github.event.inputs.target_rps || '100' }} | |
| -Dtest=SoakIntegrationTest | |
| -Dsurefire.failIfNoSpecifiedTests=false | |
| - name: Upload surefire reports on failure | |
| if: failure() | |
| uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 | |
| with: | |
| name: surefire-reports-soak | |
| path: cycles-protocol-service/cycles-protocol-service-api/target/surefire-reports/ | |
| retention-days: 14 |