The raw bucket calitp-gtfs-rt-raw-v2 is ~44 TiB spread across ~2.23 billion tiny
(20 KB) objects on Standard storage ($1,036/mo). We want to collapse those into a
small number of large files in Archive-class storage. Before committing to a
full-backlog job, try a couple of approaches on a sample and compare cost, speed,
and effort.
Try a few options on a slice of the data (e.g. one day of one feed_type), for
example:
- A compaction job (Cloud Run job / Dataflow / Airflow) that reads the tiny objects
for a period and writes one large file — e.g. Avro, or gzipped-concatenated
protobuf — directly to an Archive-class bucket.
- Different formats / rollup granularity (hourly vs daily files).
For each, note how long it took, the rough cost, and how hard it was.
Done when:
- We have a recommended approach with measured time + cost to compact a sample.
- The compacted Archive files are still readable/reprocessable —
parse and
validate can read the data back out — so nothing is lost in the format change.
- We have enough to size the full-backlog job.
The raw bucket
calitp-gtfs-rt-raw-v2is ~44 TiB spread across ~2.23 billion tiny(
20 KB) objects on Standard storage ($1,036/mo). We want to collapse those into asmall number of large files in Archive-class storage. Before committing to a
full-backlog job, try a couple of approaches on a sample and compare cost, speed,
and effort.
Try a few options on a slice of the data (e.g. one day of one feed_type), for
example:
for a period and writes one large file — e.g. Avro, or gzipped-concatenated
protobuf — directly to an Archive-class bucket.
For each, note how long it took, the rough cost, and how hard it was.
Done when:
parseandvalidatecan read the data back out — so nothing is lost in the format change.