During a customer update, we noticed that the OxQL queries used during normal health checks returned no data and the omdb oxql shell showed no timeseries schema with the \l command. The important bit here is that the oximeter and ClickHouse services were restarted and migrated, with the ClickHouse migration happening later.
As oximeter inserts samples, it derives a schema from them and inserts them into the database's oximeter.timeseries_schema table. However, it also maintains an in-memory cache of these here. It only inserts a new schema if it doesn't already exist in the cache, which obviously prevents inserting into a new database in this case. The table in ClickHouse is completely empty, but the in-memory cache is populated.
The in-memory cache is an optimization, to prevent the lookups of the existing schema each time we insert new data. But keeping these coherent is hard. We need to either ensure we somehow sync this with the database, or we remove the cache entirely and only use the database itself. The latter is certainly easier.
During a customer update, we noticed that the OxQL queries used during normal health checks returned no data and the
omdb oxqlshell showed no timeseries schema with the\lcommand. The important bit here is that theoximeterand ClickHouse services were restarted and migrated, with the ClickHouse migration happening later.As
oximeterinserts samples, it derives a schema from them and inserts them into the database'soximeter.timeseries_schematable. However, it also maintains an in-memory cache of these here. It only inserts a new schema if it doesn't already exist in the cache, which obviously prevents inserting into a new database in this case. The table in ClickHouse is completely empty, but the in-memory cache is populated.The in-memory cache is an optimization, to prevent the lookups of the existing schema each time we insert new data. But keeping these coherent is hard. We need to either ensure we somehow sync this with the database, or we remove the cache entirely and only use the database itself. The latter is certainly easier.