Risk of exceeding BQ partition limits

[mattmathis]

There is a 4000 partition limit for BQ tables, our unified views are already above 3950 days. It is not clear if the table partition limit applies to views.

Note also that our oldest web100 data is from 2009-02-18, which is more than 4100 days in the past:
SELECT min(partition_date), date_diff(current_date(), min(partition_date), day) FROM measurement-lab.ndt.web100
(I have not yet explored the 50+ day discrepancy)

We need a long term plan going forward:

• Ask to raise the limit
• partition by weeks
• partition on other units. e.g. server-months or server-years
• Use named tables for years, conventional partitioning for days within years
• limit unified views to 10 years back

[laiyi-ohlsen]

We might have dodged a bullet?

[mattmathis]

This is a non-problem - legacy tables are not growing, and the new tables are column partitioned, which does not have this limit.