Skip to content

Task: Create Batch Job to Run Aggregation Query and Update hit_count recordsΒ #357

Description

@nelsonic

As noted in #355 the aggregation query (SELECT COUNT(*) FROM hits WHERE repo_id = 123) πŸ’­
takes 2 MINUTES on a table with 30 Million Rows. ⏳
And is the reason the request/query time-out and crashes the server! πŸ’”

So we're implementing hit_count πŸ†•
which is just an integer counter 1️⃣
to return millisecond responses again. 🏎️ 🏁 πŸŽ‰

The problem is that we are temporarily ignoring the historical data in the hits table ... πŸ™„
But the data is not "lost" it's just in stasis. πŸ’­
So ...

Todo

  • Create a batch job to go through the hit_count table
    and update count to reflect the aggregate from hits. πŸ§‘β€πŸ’»
  • This must be done gradually to avoid crashing the server again. βš–οΈ
  • Once all records are updated, we can sunset this data migration script. πŸŒ†
  • I expect writing the script to take around T1h and running it to take T1w ⏳
    (Running the script once per minute to process a single record. 1440 runs per day. 10,000/week)

Will work on this immediately after I have #355 working.

Metadata

Metadata

Assignees

Type

Projects

Status
More ToDo ThanCanEver Be Done

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions