|
| 1 | +# tools/wmata_capture |
| 2 | + |
| 3 | +Capture live WMATA bus data into the exact fixture format the |
| 4 | +`transitclockIntegration` suite already ingests, so we can refresh the |
| 5 | +aging 2016 fixtures with fresh, realistic traces. |
| 6 | + |
| 7 | +The capture is a single-file Python script (`capture.py`) with |
| 8 | +[PEP 723](https://peps.python.org/pep-0723/) inline dependency metadata. |
| 9 | +It's designed to run for hours in the background and produce: |
| 10 | + |
| 11 | +1. `<output-dir>/gtfs/` — an unpacked static GTFS feed (the feed WMATA |
| 12 | + publishes at the instant you start the capture). |
| 13 | +2. `<output-dir>/avl/<route_id>_<vehicle_id>.csv` — one CSV per vehicle, |
| 14 | + formatted for `BatchCsvAvlFeedModule` with header |
| 15 | + `vehicleId,time,assignmentId,assignmentType,heading,latitude,longitude`. |
| 16 | +3. `<output-dir>/capture.log` — run log (appended across repeated runs |
| 17 | + against the same output dir). |
| 18 | + |
| 19 | +## Why this exists |
| 20 | + |
| 21 | +Two integration tests in `transitclockIntegration` are currently |
| 22 | +`@Ignore`'d because their 2016 WMATA fixtures have aged past the point |
| 23 | +where the pinned assertions are meaningful — see |
| 24 | +[OneBusAway/thetransitclock#7](https://github.com/OneBusAway/thetransitclock/issues/7) |
| 25 | +and |
| 26 | +[OneBusAway/thetransitclock#8](https://github.com/OneBusAway/thetransitclock/issues/8). |
| 27 | +To re-enable them properly we need fresh AVL traces and a fresh baseline |
| 28 | +prediction CSV. This tool produces the AVL side; the baseline CSV is then |
| 29 | +regenerated by re-running the prediction test against the captured data. |
| 30 | + |
| 31 | +## Prerequisites |
| 32 | + |
| 33 | +- `uv` installed (`brew install uv` or see |
| 34 | + [astral.sh/uv](https://docs.astral.sh/uv/)). |
| 35 | +- A WMATA developer API key from |
| 36 | + [developer.wmata.com](https://developer.wmata.com). The free tier is |
| 37 | + sufficient. |
| 38 | +- `WMATA_API_KEY` available to the script. Copy `.env.example` to `.env` |
| 39 | + and drop your key in — the script auto-loads `tools/wmata_capture/.env` |
| 40 | + at startup, so you don't need `source .env` or any shell gymnastics. |
| 41 | + If you prefer an explicit `export WMATA_API_KEY=...`, that still wins |
| 42 | + over the file. **Never** commit the key, and never pass it as a CLI |
| 43 | + flag (it would show up in shell history and in `ps` output). |
| 44 | + |
| 45 | +The script declares its own Python dependencies (`requests`, |
| 46 | +`gtfs-realtime-bindings`) via PEP 723, so `uv run capture.py ...` will |
| 47 | +resolve them into an ephemeral venv the first time. |
| 48 | + |
| 49 | +## Run recipes |
| 50 | + |
| 51 | +### Quick smoke test (5 minutes, one route) |
| 52 | + |
| 53 | +```sh |
| 54 | +# .env is auto-loaded from tools/wmata_capture/.env — no `source` needed. |
| 55 | +uv run capture.py \ |
| 56 | + --output-dir ./output/smoke-$(date +%Y%m%d-%H%M%S) \ |
| 57 | + --duration-hours 0.083 \ |
| 58 | + --routes S2 |
| 59 | +``` |
| 60 | + |
| 61 | +Expected output: |
| 62 | + |
| 63 | +- `./output/smoke-.../gtfs/trips.txt` etc. present |
| 64 | +- `./output/smoke-.../avl/S2_<vehicleId>.csv` for every S2 vehicle the |
| 65 | + feed reported |
| 66 | +- `./output/smoke-.../capture.log` with `polls ok: ~10` and a row count |
| 67 | + |
| 68 | +### Realistic fixture capture (AM rush, small route set) |
| 69 | + |
| 70 | +```sh |
| 71 | +uv run capture.py \ |
| 72 | + --output-dir ./output/fixtures-$(date +%Y%m%d) \ |
| 73 | + --duration-hours 4 \ |
| 74 | + --poll-interval 30 \ |
| 75 | + --routes S2,3T,5A |
| 76 | +``` |
| 77 | + |
| 78 | +Run against Eastern weekday AM peak (~06:00–10:00 ET) for the most data |
| 79 | +per vehicle. Background it with `nohup ... &` or run under `tmux`. |
| 80 | + |
| 81 | +### Single-vehicle targeted capture |
| 82 | + |
| 83 | +```sh |
| 84 | +uv run capture.py \ |
| 85 | + --output-dir ./output/target-$(date +%Y%m%d) \ |
| 86 | + --duration-hours 2 \ |
| 87 | + --vehicles 2113 |
| 88 | +``` |
| 89 | + |
| 90 | +## Promoting captures to fixtures |
| 91 | + |
| 92 | +Once a capture looks good: |
| 93 | + |
| 94 | +1. Replace the module's GTFS: |
| 95 | + ```sh |
| 96 | + rm -rf transitclockIntegration/src/test/resources/gtfs/S2 |
| 97 | + cp -R tools/wmata_capture/output/<run>/gtfs transitclockIntegration/src/test/resources/gtfs/S2 |
| 98 | + ``` |
| 99 | + Adjust the target subdir name to the route the test uses. |
| 100 | +2. Replace the AVL CSV: |
| 101 | + ```sh |
| 102 | + cp tools/wmata_capture/output/<run>/avl/S2_<vehicleId>.csv \ |
| 103 | + transitclockIntegration/src/test/resources/avl/S2_<vehicleId>.csv |
| 104 | + ``` |
| 105 | + Rename to match the test's hard-coded constant |
| 106 | + (e.g. `S2_2113.csv`). |
| 107 | +3. **Regenerate the prediction baseline** (`pred/*.csv`). This is not |
| 108 | + produced by this capture tool — it's the *output* of running the |
| 109 | + current predictor against the new AVL. To rebuild: |
| 110 | + - Temporarily remove the `@Ignore` from |
| 111 | + `PredictionAccuracyIntegrationTest`. |
| 112 | + - Run the test; dump the `new`-side predictions (see the test's |
| 113 | + `setUp` — `session.createCriteria(Prediction.class).list()`) to a |
| 114 | + CSV with the same columns as the existing |
| 115 | + `pred/S2_2113.csv`. |
| 116 | + - Save that CSV as the new baseline. |
| 117 | + - Leave the `@Ignore` removed — un-ignoring the test permanently is |
| 118 | + the whole point of the refresh. Don't re-add `@Ignore` after |
| 119 | + generating the baseline. |
| 120 | +4. Commit the new fixtures + baseline in one PR; reference issues |
| 121 | + OneBusAway/thetransitclock#7 and #8 in the description. |
| 122 | + |
| 123 | +## Known caveats |
| 124 | + |
| 125 | +- **`block_id` in WMATA GTFS is not always populated.** The script falls |
| 126 | + back to `assignmentType=TRIP_ID` when the GTFS-RT feed reports a |
| 127 | + `trip_id` that has no corresponding `block_id` in `trips.txt`. |
| 128 | + `BatchCsvAvlFeedModule` + `BlockAssigner` handle both types, but some |
| 129 | + transitclock behavior paths depend on block assignment specifically — |
| 130 | + prefer capturing a route/period where `trips.txt` does carry block_ids |
| 131 | + (check via `awk -F, '{print $<block-col>}' trips.txt | sort -u | head`). |
| 132 | +- **The GTFS-RT feed repeats observations.** The script de-duplicates on |
| 133 | + `(vehicle_id, feed-reported timestamp)` within a single run, so a |
| 134 | + 30-second poll interval against a feed that only updates every 60s |
| 135 | + won't double up. De-dup state is not persisted across runs — don't |
| 136 | + resume into the same output dir and expect continuity; start a new |
| 137 | + output dir. |
| 138 | +- **Clock.** Timestamps are emitted in `America/New_York` local time |
| 139 | + formatted `MM-dd-yyyy HH:mm:ss` (the format `BatchCsvAvlFeedModule` |
| 140 | + parses using the JVM default timezone). The capture host's TZ does |
| 141 | + not matter. |
| 142 | +- **Outages happen.** WMATA's API can return 5xx and can stall for |
| 143 | + minutes. The script retries with a bounded exponential backoff (up to |
| 144 | + ~2 minutes between attempts) rather than dying. |
| 145 | + |
| 146 | +## Files |
| 147 | + |
| 148 | +- `capture.py` — the script (PEP 723, single file). |
| 149 | +- `.env.example` — template for `WMATA_API_KEY`. Real `.env` is |
| 150 | + gitignored. |
| 151 | +- `output/` — default capture destination. Gitignored. |
0 commit comments