Integration tests

.github/workflows/ci.yml

@@ -38,6 +38,13 @@ jobs:
       - name: Run pipeline tests
         run: mvn -B -ntp -pl transitclockPipelineTests -am -P include-pipeline-tests test
 
+      # Integration tests live in transitclockIntegration and are the
+      # heaviest tier — full AVL-CSV replays through a booted Core. Opt-in
+      # via include-integration-tests. Kept as its own step so a failure
+      # here is attributed distinctly from the unit and pipeline tiers.
+      - name: Run integration tests
+        run: mvn -B -ntp -pl transitclockIntegration -am -P include-integration-tests test
+
       - name: Upload JaCoCo aggregate report
         if: always()
         uses: actions/upload-artifact@v4

.gitignore

@@ -13,3 +13,9 @@
 .DS_Store
 
 settings.local.json
+
+# WMATA capture output — live API data, not fixture-ready, not for VCS
+/tools/wmata_capture/output/
+/tools/wmata_capture/.env
+
+__pycache__/

CLAUDE.md

@@ -16,6 +16,7 @@ Built from the repository root as a Maven multi-module project.
 - Run a single test class: `mvn -pl transitclock test -Dtest=TestAPIKeyManager`
 - Integration tests live in the `transitclockIntegration` module and are **excluded by default** via the `skip-integration-tests` profile. Enable with: `mvn install -P include-integration-tests`
 - Pipeline tests live in the `transitclockPipelineTests` module and are **excluded by default** — opt-in via the `include-pipeline-tests` profile. They boot a real Core against an in-memory HSQL database populated with a small WMATA GTFS fixture and exercise matcher/generator behavior end-to-end (lighter than the full AVL-trace runs in `transitclockIntegration`). Enable with: `mvn -pl transitclockPipelineTests -am -P include-pipeline-tests test`. CI runs this as a separate step after `mvn verify`.
+- Run **everything** (unit + pipeline + integration) in one invocation: `mvn install -P run-all-tests`. The `run-all-tests` profile activates both extra modules so the reactor picks them up alongside the default build.
 
 `mvn test` on the full reactor fails: `transitclockQuickStart` binds `maven-dependency-plugin:copy` to `generate-resources` to pull the `transitclockApi` WAR into its resources, but the `test` phase never packages that WAR (MDEP-187: "Artifact has not been packaged yet"). Use `mvn verify` / `mvn package` / `mvn install` to exercise all tests, or scope to one module with `-pl`, or skip QuickStart: `mvn test -pl '!transitclockQuickStart'`.
 - Shaded executable JARs are emitted into `transitclock/target/` (e.g. `Core.jar`, `GtfsFileProcessor.jar`, `SchemaGenerator.jar`, `CreateWebAgency.jar`, `CreateAPIKey.jar`, `RmiQuery.jar`, `UpdateTravelTimes.jar`, `ScheduleGenerator.jar`) — each is a maven-shade execution in `transitclock/pom.xml`.

README.md

@@ -57,6 +57,11 @@ Two additional test suites are opt-in via Maven profiles and excluded from the d
   mvn install -P include-integration-tests
   ```
 
+To run **everything** (unit + pipeline + integration) in one go:
+```
+mvn install -P run-all-tests
+```
+
 ### Code coverage
 
 JaCoCo generates coverage reports as part of the Maven `verify` phase.

pom.xml

@@ -68,5 +68,12 @@
                 <module>transitclockPipelineTests</module>
             </modules>
         </profile>
+        <profile>
+            <id>run-all-tests</id>
+            <modules>
+                <module>transitclockIntegration</module>
+                <module>transitclockPipelineTests</module>
+            </modules>
+        </profile>
     </profiles>
 </project>

tools/wmata_capture/.env.example

@@ -0,0 +1,12 @@
+# Copy this file to `.env` and fill in your real key. `.env` is gitignored.
+#
+# The capture script loads this file automatically at startup — you don't
+# need `source .env` or any shell gymnastics. An explicit
+# `export WMATA_API_KEY=...` in your shell still takes precedence.
+#
+# Request a key at https://developer.wmata.com — the free tier covers a
+# handful of consecutive hours of GTFS-RT polling, which is plenty to
+# capture fixture-sized traces (~4h).
+#
+# DO NOT commit your real key. DO NOT pass it as a CLI flag.
+WMATA_API_KEY=your-wmata-api-key-here

tools/wmata_capture/README.md

@@ -0,0 +1,151 @@
+# tools/wmata_capture
+
+Capture live WMATA bus data into the exact fixture format the
+`transitclockIntegration` suite already ingests, so we can refresh the
+aging 2016 fixtures with fresh, realistic traces.
+
+The capture is a single-file Python script (`capture.py`) with
+[PEP 723](https://peps.python.org/pep-0723/) inline dependency metadata.
+It's designed to run for hours in the background and produce:
+
+1. `<output-dir>/gtfs/` — an unpacked static GTFS feed (the feed WMATA
+   publishes at the instant you start the capture).
+2. `<output-dir>/avl/<route_id>_<vehicle_id>.csv` — one CSV per vehicle,
+   formatted for `BatchCsvAvlFeedModule` with header
+   `vehicleId,time,assignmentId,assignmentType,heading,latitude,longitude`.
+3. `<output-dir>/capture.log` — run log (appended across repeated runs
+   against the same output dir).
+
+## Why this exists
+
+Two integration tests in `transitclockIntegration` are currently
+`@Ignore`'d because their 2016 WMATA fixtures have aged past the point
+where the pinned assertions are meaningful — see
+[OneBusAway/thetransitclock#7](https://github.com/OneBusAway/thetransitclock/issues/7)
+and
+[OneBusAway/thetransitclock#8](https://github.com/OneBusAway/thetransitclock/issues/8).
+To re-enable them properly we need fresh AVL traces and a fresh baseline
+prediction CSV. This tool produces the AVL side; the baseline CSV is then
+regenerated by re-running the prediction test against the captured data.
+
+## Prerequisites
+
+- `uv` installed (`brew install uv` or see
+  [astral.sh/uv](https://docs.astral.sh/uv/)).
+- A WMATA developer API key from
+  [developer.wmata.com](https://developer.wmata.com). The free tier is
+  sufficient.
+- `WMATA_API_KEY` available to the script. Copy `.env.example` to `.env`
+  and drop your key in — the script auto-loads `tools/wmata_capture/.env`
+  at startup, so you don't need `source .env` or any shell gymnastics.
+  If you prefer an explicit `export WMATA_API_KEY=...`, that still wins
+  over the file. **Never** commit the key, and never pass it as a CLI
+  flag (it would show up in shell history and in `ps` output).
+
+The script declares its own Python dependencies (`requests`,
+`gtfs-realtime-bindings`) via PEP 723, so `uv run capture.py ...` will
+resolve them into an ephemeral venv the first time.
+
+## Run recipes
+
+### Quick smoke test (5 minutes, one route)
+
+```sh
+# .env is auto-loaded from tools/wmata_capture/.env — no `source` needed.
+uv run capture.py \
+    --output-dir ./output/smoke-$(date +%Y%m%d-%H%M%S) \
+    --duration-hours 0.083 \
+    --routes S2
+```
+
+Expected output:
+
+- `./output/smoke-.../gtfs/trips.txt` etc. present
+- `./output/smoke-.../avl/S2_<vehicleId>.csv` for every S2 vehicle the
+  feed reported
+- `./output/smoke-.../capture.log` with `polls ok: ~10` and a row count
+
+### Realistic fixture capture (AM rush, small route set)
+
+```sh
+uv run capture.py \
+    --output-dir ./output/fixtures-$(date +%Y%m%d) \
+    --duration-hours 4 \
+    --poll-interval 30 \
+    --routes S2,3T,5A
+```
+
+Run against Eastern weekday AM peak (~06:00–10:00 ET) for the most data
+per vehicle. Background it with `nohup ... &` or run under `tmux`.
+
+### Single-vehicle targeted capture
+
+```sh
+uv run capture.py \
+    --output-dir ./output/target-$(date +%Y%m%d) \
+    --duration-hours 2 \
+    --vehicles 2113
+```
+
+## Promoting captures to fixtures
+
+Once a capture looks good:
+
+1. Replace the module's GTFS:
+   ```sh
+   rm -rf transitclockIntegration/src/test/resources/gtfs/S2
+   cp -R tools/wmata_capture/output/<run>/gtfs transitclockIntegration/src/test/resources/gtfs/S2
+   ```
+   Adjust the target subdir name to the route the test uses.
+2. Replace the AVL CSV:
+   ```sh
+   cp tools/wmata_capture/output/<run>/avl/S2_<vehicleId>.csv \
+      transitclockIntegration/src/test/resources/avl/S2_<vehicleId>.csv
+   ```
+   Rename to match the test's hard-coded constant
+   (e.g. `S2_2113.csv`).
+3. **Regenerate the prediction baseline** (`pred/*.csv`). This is not
+   produced by this capture tool — it's the *output* of running the
+   current predictor against the new AVL. To rebuild:
+   - Temporarily remove the `@Ignore` from
+     `PredictionAccuracyIntegrationTest`.
+   - Run the test; dump the `new`-side predictions (see the test's
+     `setUp` — `session.createCriteria(Prediction.class).list()`) to a
+     CSV with the same columns as the existing
+     `pred/S2_2113.csv`.
+   - Save that CSV as the new baseline.
+   - Leave the `@Ignore` removed — un-ignoring the test permanently is
+     the whole point of the refresh. Don't re-add `@Ignore` after
+     generating the baseline.
+4. Commit the new fixtures + baseline in one PR; reference issues
+   OneBusAway/thetransitclock#7 and #8 in the description.
+
+## Known caveats
+
+- **`block_id` in WMATA GTFS is not always populated.** The script falls
+  back to `assignmentType=TRIP_ID` when the GTFS-RT feed reports a
+  `trip_id` that has no corresponding `block_id` in `trips.txt`.
+  `BatchCsvAvlFeedModule` + `BlockAssigner` handle both types, but some
+  transitclock behavior paths depend on block assignment specifically —
+  prefer capturing a route/period where `trips.txt` does carry block_ids
+  (check via `awk -F, '{print $<block-col>}' trips.txt | sort -u | head`).
+- **The GTFS-RT feed repeats observations.** The script de-duplicates on
+  `(vehicle_id, feed-reported timestamp)` within a single run, so a
+  30-second poll interval against a feed that only updates every 60s
+  won't double up. De-dup state is not persisted across runs — don't
+  resume into the same output dir and expect continuity; start a new
+  output dir.
+- **Clock.** Timestamps are emitted in `America/New_York` local time
+  formatted `MM-dd-yyyy HH:mm:ss` (the format `BatchCsvAvlFeedModule`
+  parses using the JVM default timezone). The capture host's TZ does
+  not matter.
+- **Outages happen.** WMATA's API can return 5xx and can stall for
+  minutes. The script retries with a bounded exponential backoff (up to
+  ~2 minutes between attempts) rather than dying.
+
+## Files
+
+- `capture.py` — the script (PEP 723, single file).
+- `.env.example` — template for `WMATA_API_KEY`. Real `.env` is
+  gitignored.
+- `output/` — default capture destination. Gitignored.