Archive

The archiver's main responsibility is archiving both real time and inference messages. Additionally:

• The archiver picks up messages on both the Real-Time Queue and the Inference Queue.
• The archiver queries the TDS for some additional records
• The archiver does a minimal amount of data manipulation and, using Hibernate, inserts location and inference messages into their respective databases.
• The archiver hosts an Internal Operational Developer API to the most recent vehicle locations.

Design

Real-Time and Inference Archiving

Import info from Confluence

TDM Data Archiving

• Crew assignment, depot assignment, and vehicle pullout data archived daily at 4am (this is when the last updated values for the previous day come through)
• Bundles are archived by convention on S3
• DSC data is not archived (original requirement was waived)

Crew Assignment

+--------------+--------------+------+-----+---------+-------+
| Field        | Type         | Null | Key | Default | Extra |
+--------------+--------------+------+-----+---------+-------+
| id           | int(11)      | NO   | PRI | NULL    |       |
| agency_id    | varchar(64)  | YES  |     | NULL    |       |
| depot_id     | varchar(16)  | YES  |     | NULL    |       |
| operator_id  | varchar(16)  | YES  |     | NULL    |       |
| run          | varchar(255) | YES  |     | NULL    |       |
| service_date | date         | NO   |     | NULL    |       |
| updated      | datetime     | YES  |     | NULL    |       |
+--------------+--------------+------+-----+---------+-------+

Depot Assignment

+------------+-------------+------+-----+---------+-------+
| Field      | Type        | Null | Key | Default | Extra |
+------------+-------------+------+-----+---------+-------+
| id         | int(11)     | NO   | PRI | NULL    |       |
| agency_id  | varchar(64) | YES  |     | NULL    |       |
| date       | date        | NO   |     | NULL    |       |
| depot_id   | varchar(16) | YES  |     | NULL    |       |
| vehicle_id | int(11)     | NO   |     | NULL    |       |
+------------+-------------+------+-----+---------+-------+

Pullouts

+----------------+--------------+------+-----+---------+-------+
| Field          | Type         | Null | Key | Default | Extra |
+----------------+--------------+------+-----+---------+-------+
| id             | int(11)      | NO   | PRI | NULL    |       |
| agency_id      | varchar(64)  | YES  |     | NULL    |       |
| agency_id_tcip | int(11)      | YES  |     | NULL    |       |
| depot_id       | varchar(16)  | YES  |     | NULL    |       |
| operator_id    | varchar(16)  | YES  |     | NULL    |       |
| pullin_time    | datetime     | YES  |     | NULL    |       |
| pullout_time   | datetime     | YES  |     | NULL    |       |
| run            | varchar(255) | YES  |     | NULL    |       |
| service_date   | date         | NO   |     | NULL    |       |
| vehicle_id     | int(11)      | NO   |     | NULL    |       |
+----------------+--------------+------+-----+---------+-------+

Schema Change Strategies

1. Create slave database from backup of master
2. Peform upgrade modifications to database
3. Stop monitoring, turn on maintenance mode
4. Stop external data capture cron jobs
5. Stop service alerts cron job
6. Stop archiver process (tomcat6)
7. Backup master database
8. Switch CNAME from master to slave
9. Start archiver
10. Start service alerts cron job
11. Start external data capture cron jobs
12. Turn on monitoring (once system stabilizes)
13. Perform manual backfill for time from last backup to CNAME swap
14. Delete old master