Skip to content

Commit f8e8a78

Browse files
stephen-shelbyclaude
stephen-shelby
and
claude
authored
[Enhancement] Add opt-in lazy refresh of external stats cache on journal replay (#74371)
Signed-off-by: stephen <91597003+stephen-shelby@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 725fdae commit f8e8a78

14 files changed

Lines changed: 348 additions & 24 deletions

File tree

‎docs/en/administration/management/FE_parameters/user_query_loading.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -814,6 +814,15 @@ Starting from version 3.3.0, the system defaults to refreshing one partition at
814814
- Description: The interval at which the cache of statistical information is updated.
815815
- Introduced in: -
816816

817+
818+
### `enable_external_stats_lazy_refresh_on_replay`
819+
820+
- Default: false
821+
- Type: Boolean
822+
- Unit: -
823+
- Is mutable: Yes
824+
- Description: Controls how followers (and restart recovery) refresh the connector (external table) statistics cache when replaying statistics journals. When set to `true`, the cache is invalidated by the table UUID persisted in the journal and reloaded lazily on the next query, which avoids resolving external table metadata (`MetadataMgr.getTable`) during replay — such resolution may block the journal replayer on the Hive Metastore or object storage. When set to `false` (default), the legacy eager refresh is used, preserving existing behavior. Statistics journals written before this UUID was persisted always fall back to eager refresh regardless of this setting.
825+
817826
### `statistics_large_string_column_merge_threshold`
818827

819828
- Default: 0

‎docs/ja/administration/management/FE_parameters/user_query_loading.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -805,6 +805,15 @@ ADMIN SET FRONTEND CONFIG ("key" = "value");
805805
- 説明：統計情報のキャッシュが更新される間隔。
806806
- 導入時期：-
807807

808+
809+
### `enable_external_stats_lazy_refresh_on_replay`
810+
811+
- デフォルト：false
812+
- タイプ：Boolean
813+
- 単位：-
814+
- 変更可能：Yes
815+
- 説明：統計情報ジャーナルのリプレイ時に、Follower（および再起動リカバリ）が Connector（外部テーブル）統計情報キャッシュをどのようにリフレッシュするかを制御します。`true` の場合、ジャーナルに永続化されたテーブル UUID でキャッシュを無効化し、次回クエリ時に遅延ロードします。これにより、リプレイ中の外部テーブルメタデータ解決（`MetadataMgr.getTable`）を回避できます。この解決は Hive Metastore やオブジェクトストレージが遅い場合にジャーナルリプレイスレッドをブロックする可能性があります。`false`（デフォルト）の場合は従来の積極的なリフレッシュを使用し、既存の動作を維持します。この UUID が永続化される前に書き込まれた統計情報ジャーナルは、この設定に関係なく常に積極的なリフレッシュにフォールバックします。
816+
808817
### `statistics_large_string_column_merge_threshold`
809818

810819
- デフォルト：0

‎docs/zh/administration/management/FE_parameters/user_query_loading.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -814,6 +814,15 @@ ADMIN SET FRONTEND CONFIG ("key" = "value");
814814
- 描述: 统计信息缓存的更新间隔。
815815
- 引入版本: -
816816

817+
818+
### `enable_external_stats_lazy_refresh_on_replay`
819+
820+
- 默认值: false
821+
- 类型: Boolean
822+
- 单位: -
823+
- 是否可变: Yes
824+
- 描述: 控制 Follower（以及重启恢复）在回放统计信息日志时如何刷新 Connector（外部表）统计信息缓存。设为 `true` 时，按日志中持久化的表 UUID 失效缓存，并在下次查询时惰性重新加载，从而避免在回放期间解析外部表元数据（`MetadataMgr.getTable`）——该解析可能因 Hive Metastore 或对象存储变慢而阻塞日志回放线程。设为 `false`（默认）时使用原有的主动刷新方式，保持现有行为。在该 UUID 被持久化之前写入的统计信息日志，无论该配置如何都会回退到主动刷新。
825+
817826
### `statistics_large_string_column_merge_threshold`
818827

819828
- 默认值: 0

‎fe/fe-core/src/main/java/com/starrocks/common/Config.java‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2558,6 +2558,14 @@ public class Config extends ConfigBase {
25582558
@ConfField(mutable = true)
25592559
public static boolean enable_statistic_cache_refresh_after_write = false;
25602560

2561+
@ConfField(mutable = true, comment = "When replaying external-table statistics journals on followers " +
2562+
"(and during restart recovery), invalidate the connector statistics cache by the table UUID " +
2563+
"persisted in the journal and reload it lazily on the next query, instead of eagerly refreshing " +
2564+
"it. Eager refresh resolves external table metadata (MetadataMgr.getTable), which may block the " +
2565+
"replayer thread on HMS/object storage. Disabled by default to preserve the legacy eager-refresh " +
2566+
"behavior; journals without a persisted UUID always fall back to eager refresh regardless of this flag.")
2567+
public static boolean enable_external_stats_lazy_refresh_on_replay = false;
2568+
25612569
@ConfField(mutable = true)
25622570
public static long statistic_collect_too_many_version_sleep = 600000; // 10min
25632571
/**

‎fe/fe-core/src/main/java/com/starrocks/persist/EditLog.java‎

Lines changed: 10 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -985,50 +985,38 @@ public void loadJournal(GlobalStateMgr globalStateMgr, JournalEntity journal)
985985
case OperationType.OP_ADD_EXTERNAL_BASIC_STATS_META: {
986986
ExternalBasicStatsMeta basicStatsMeta = (ExternalBasicStatsMeta) journal.data();
987987
globalStateMgr.getAnalyzeMgr().replayAddExternalBasicStatsMeta(basicStatsMeta);
988-
// The follower replays the stats meta log, indicating that the master has re-completed
989-
// statistic, and the follower's should refresh cache here.
990-
// We don't need to refresh statistics when checkpointing
988+
// The follower replays the stats meta log, indicating that the leader has re-completed
989+
// statistics, so the follower should drop its connector stats cache here and reload it
990+
// lazily on the next query. We don't need to touch the cache when checkpointing.
991991
if (!GlobalStateMgr.isCheckpointThread()) {
992-
globalStateMgr.getAnalyzeMgr().refreshConnectorTableBasicStatisticsCache(
993-
basicStatsMeta.getCatalogName(),
994-
basicStatsMeta.getDbName(), basicStatsMeta.getTableName(),
995-
basicStatsMeta.getColumns(), true);
992+
globalStateMgr.getAnalyzeMgr().replayRefreshExternalBasicStatsCache(basicStatsMeta);
996993
}
997994
break;
998995
}
999996
case OperationType.OP_REMOVE_EXTERNAL_BASIC_STATS_META: {
1000997
ExternalBasicStatsMeta basicStatsMeta = (ExternalBasicStatsMeta) journal.data();
1001998
globalStateMgr.getAnalyzeMgr().replayRemoveExternalBasicStatsMeta(basicStatsMeta);
1002999
if (!GlobalStateMgr.isCheckpointThread()) {
1003-
globalStateMgr.getAnalyzeMgr().expireConnectorTableAndColumnStatistics(
1004-
basicStatsMeta.getCatalogName(),
1005-
basicStatsMeta.getDbName(), basicStatsMeta.getTableName(),
1006-
basicStatsMeta.getColumns());
1000+
globalStateMgr.getAnalyzeMgr().replayExpireExternalBasicStatsCache(basicStatsMeta);
10071001
}
10081002
break;
10091003
}
10101004
case OperationType.OP_ADD_EXTERNAL_HISTOGRAM_STATS_META: {
10111005
ExternalHistogramStatsMeta histogramStatsMeta = (ExternalHistogramStatsMeta) journal.data();
10121006
globalStateMgr.getAnalyzeMgr().replayAddExternalHistogramStatsMeta(histogramStatsMeta);
1013-
// The follower replays the stats meta log, indicating that the master has re-completed
1014-
// statistic, and the follower's should expire cache here.
1015-
// We don't need to refresh statistics when checkpointing
1007+
// The follower replays the stats meta log, indicating that the leader has re-completed
1008+
// statistics, so the follower should drop its connector histogram cache here and reload it
1009+
// lazily on the next query. We don't need to touch the cache when checkpointing.
10161010
if (!GlobalStateMgr.isCheckpointThread()) {
1017-
globalStateMgr.getAnalyzeMgr().refreshConnectorTableHistogramStatisticsCache(
1018-
histogramStatsMeta.getCatalogName(), histogramStatsMeta.getDbName(),
1019-
histogramStatsMeta.getTableName(),
1020-
Lists.newArrayList(histogramStatsMeta.getColumn()), true);
1011+
globalStateMgr.getAnalyzeMgr().replayRefreshExternalHistogramStatsCache(histogramStatsMeta);
10211012
}
10221013
break;
10231014
}
10241015
case OperationType.OP_REMOVE_EXTERNAL_HISTOGRAM_STATS_META: {
10251016
ExternalHistogramStatsMeta histogramStatsMeta = (ExternalHistogramStatsMeta) journal.data();
10261017
globalStateMgr.getAnalyzeMgr().replayRemoveExternalHistogramStatsMeta(histogramStatsMeta);
10271018
if (!GlobalStateMgr.isCheckpointThread()) {
1028-
globalStateMgr.getAnalyzeMgr().expireConnectorTableHistogramStatisticsCache(
1029-
histogramStatsMeta.getCatalogName(), histogramStatsMeta.getDbName(),
1030-
histogramStatsMeta.getTableName(),
1031-
Lists.newArrayList(histogramStatsMeta.getColumn()));
1019+
globalStateMgr.getAnalyzeMgr().replayExpireExternalHistogramStatsCache(histogramStatsMeta);
10321020
}
10331021
break;
10341022
}

‎fe/fe-core/src/main/java/com/starrocks/sql/optimizer/statistics/CachedStatisticStorage.java‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -306,6 +306,17 @@ public void expireConnectorTableColumnStatistics(Table table, List<String> colum
306306
connectorTableCachedStatistics.synchronous().invalidateAll(allKeys);
307307
}
308308

309+
@Override
310+
public void invalidateConnectorTableColumnStatistics(String tableUUID, List<String> columns) {
311+
if (tableUUID == null || tableUUID.isEmpty() || columns == null) {
312+
return;
313+
}
314+
List<ConnectorTableColumnKey> allKeys = columns.stream()
315+
.map(column -> new ConnectorTableColumnKey(tableUUID, column))
316+
.collect(Collectors.toList());
317+
connectorTableCachedStatistics.synchronous().invalidateAll(allKeys);
318+
}
319+
309320
@Override
310321
public void refreshConnectorTableColumnStatistics(Table table, List<String> columns, boolean isSync) {
311322
Preconditions.checkState(table != null);
@@ -634,6 +645,17 @@ public void expireConnectorHistogramStatistics(Table table, List<String> columns
634645
connectorHistogramCache.synchronous().invalidateAll(allKeys);
635646
}
636647

648+
@Override
649+
public void invalidateConnectorHistogramStatistics(String tableUUID, List<String> columns) {
650+
if (tableUUID == null || tableUUID.isEmpty() || columns == null) {
651+
return;
652+
}
653+
List<ConnectorTableColumnKey> allKeys = columns.stream()
654+
.map(column -> new ConnectorTableColumnKey(tableUUID, column))
655+
.collect(Collectors.toList());
656+
connectorHistogramCache.synchronous().invalidateAll(allKeys);
657+
}
658+
637659
private List<ColumnStatistic> getDefaultColumnStatisticList(List<String> columns) {
638660
List<ColumnStatistic> columnStatisticList = new ArrayList<>();
639661
for (int i = 0; i < columns.size(); ++i) {

‎fe/fe-core/src/main/java/com/starrocks/sql/optimizer/statistics/StatisticStorage.java‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -102,11 +102,25 @@ default void expireTableAndColumnStatistics(Table table, List<String> columns) {
102102
default void expireConnectorTableColumnStatistics(Table table, List<String> columns) {
103103
}
104104

105+
/**
106+
* Invalidate connector column statistics by the table UUID directly, without resolving the
107+
* {@link Table} (and therefore without any external metadata I/O). Used during journal replay.
108+
*/
109+
default void invalidateConnectorTableColumnStatistics(String tableUUID, List<String> columns) {
110+
}
111+
105112
default void refreshConnectorTableColumnStatistics(Table table, List<String> columns, boolean isSync) {
106113
}
107114

108115
default void expireConnectorHistogramStatistics(Table table, List<String> columns) {
109116
}
110117

118+
/**
119+
* Invalidate connector histogram statistics by the table UUID directly, without resolving the
120+
* {@link Table} (and therefore without any external metadata I/O). Used during journal replay.
121+
*/
122+
default void invalidateConnectorHistogramStatistics(String tableUUID, List<String> columns) {
123+
}
124+
111125
void addColumnStatistic(Table table, String column, ColumnStatistic columnStatistic);
112126
}

‎fe/fe-core/src/main/java/com/starrocks/statistic/AnalyzeMgr.java‎

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -498,6 +498,80 @@ public void expireConnectorTableHistogramStatisticsCache(String catalogName, Str
498498
GlobalStateMgr.getCurrentState().getStatisticStorage().expireConnectorHistogramStatistics(table, columns);
499499
}
500500

501+
/**
502+
* Replay handler for OP_ADD_EXTERNAL_BASIC_STATS_META on followers.
503+
* <p>
504+
* Prefer invalidating the connector cache by the table UUID persisted in the journal, so that replay
505+
* never resolves external table metadata (which may block on HMS/object storage and stall the replayer).
506+
* The fresh statistics are reloaded lazily on the next query. Fall back to the legacy metadata-based
507+
* refresh only for journals written before the UUID was persisted.
508+
*/
509+
public void replayRefreshExternalBasicStatsCache(ExternalBasicStatsMeta basicStatsMeta) {
510+
if (invalidateConnectorColumnStatsByUUID(basicStatsMeta.getTableUUID(), basicStatsMeta.getColumns())) {
511+
return;
512+
}
513+
refreshConnectorTableBasicStatisticsCache(basicStatsMeta.getCatalogName(), basicStatsMeta.getDbName(),
514+
basicStatsMeta.getTableName(), basicStatsMeta.getColumns(), true);
515+
}
516+
517+
/**
518+
* Replay handler for OP_REMOVE_EXTERNAL_BASIC_STATS_META on followers.
519+
* See {@link #replayRefreshExternalBasicStatsCache} for the UUID-first / legacy-fallback rationale.
520+
*/
521+
public void replayExpireExternalBasicStatsCache(ExternalBasicStatsMeta basicStatsMeta) {
522+
if (invalidateConnectorColumnStatsByUUID(basicStatsMeta.getTableUUID(), basicStatsMeta.getColumns())) {
523+
return;
524+
}
525+
expireConnectorTableAndColumnStatistics(basicStatsMeta.getCatalogName(), basicStatsMeta.getDbName(),
526+
basicStatsMeta.getTableName(), basicStatsMeta.getColumns());
527+
}
528+
529+
/**
530+
* Replay handler for OP_ADD_EXTERNAL_HISTOGRAM_STATS_META on followers.
531+
* See {@link #replayRefreshExternalBasicStatsCache} for the UUID-first / legacy-fallback rationale.
532+
*/
533+
public void replayRefreshExternalHistogramStatsCache(ExternalHistogramStatsMeta histogramStatsMeta) {
534+
List<String> columns = Lists.newArrayList(histogramStatsMeta.getColumn());
535+
if (invalidateConnectorHistogramStatsByUUID(histogramStatsMeta.getTableUUID(), columns)) {
536+
return;
537+
}
538+
refreshConnectorTableHistogramStatisticsCache(histogramStatsMeta.getCatalogName(),
539+
histogramStatsMeta.getDbName(), histogramStatsMeta.getTableName(), columns, true);
540+
}
541+
542+
/**
543+
* Replay handler for OP_REMOVE_EXTERNAL_HISTOGRAM_STATS_META on followers.
544+
* See {@link #replayRefreshExternalBasicStatsCache} for the UUID-first / legacy-fallback rationale.
545+
*/
546+
public void replayExpireExternalHistogramStatsCache(ExternalHistogramStatsMeta histogramStatsMeta) {
547+
List<String> columns = Lists.newArrayList(histogramStatsMeta.getColumn());
548+
if (invalidateConnectorHistogramStatsByUUID(histogramStatsMeta.getTableUUID(), columns)) {
549+
return;
550+
}
551+
expireConnectorTableHistogramStatisticsCache(histogramStatsMeta.getCatalogName(),
552+
histogramStatsMeta.getDbName(), histogramStatsMeta.getTableName(), columns);
553+
}
554+
555+
private boolean invalidateConnectorColumnStatsByUUID(String tableUUID, List<String> columns) {
556+
// Feature off (default) or legacy journal without a UUID: fall back to the legacy eager refresh.
557+
if (!Config.enable_external_stats_lazy_refresh_on_replay || tableUUID == null || tableUUID.isEmpty()) {
558+
return false;
559+
}
560+
GlobalStateMgr.getCurrentState().getStatisticStorage()
561+
.invalidateConnectorTableColumnStatistics(tableUUID, columns);
562+
return true;
563+
}
564+
565+
private boolean invalidateConnectorHistogramStatsByUUID(String tableUUID, List<String> columns) {
566+
// Feature off (default) or legacy journal without a UUID: fall back to the legacy eager refresh.
567+
if (!Config.enable_external_stats_lazy_refresh_on_replay || tableUUID == null || tableUUID.isEmpty()) {
568+
return false;
569+
}
570+
GlobalStateMgr.getCurrentState().getStatisticStorage()
571+
.invalidateConnectorHistogramStatistics(tableUUID, columns);
572+
return true;
573+
}
574+
501575
public void clearStatisticFromDroppedTable() {
502576
clearStatisticFromNativeDroppedTable();
503577
clearStatisticFromExternalDroppedTable();

‎fe/fe-core/src/main/java/com/starrocks/statistic/ExternalBasicStatsMeta.java‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,13 @@ public class ExternalBasicStatsMeta implements Writable {
5151
@SerializedName("columnStats")
5252
private Map<String, ColumnStatsMeta> columnStatsMetaMap = Maps.newConcurrentMap();
5353

54+
// The table UUID resolved on the leader at analyze time. It is the key used by the connector
55+
// statistics cache, so persisting it lets followers invalidate the cache during journal replay
56+
// without resolving external table metadata (which may block on HMS/object storage).
57+
// May be null for journals written before this field was introduced (see replay fallback).
58+
@SerializedName("tableUUID")
59+
private String tableUUID;
60+
5461
public ExternalBasicStatsMeta() {}
5562

5663
public ExternalBasicStatsMeta(String catalogName, String dbName, String tableName, List<String> columns,
@@ -78,6 +85,14 @@ public String getTableName() {
7885
return tableName;
7986
}
8087

88+
public String getTableUUID() {
89+
return tableUUID;
90+
}
91+
92+
public void setTableUUID(String tableUUID) {
93+
this.tableUUID = tableUUID;
94+
}
95+
8196
public List<String> getColumns() {
8297
return columns;
8398
}

‎fe/fe-core/src/main/java/com/starrocks/statistic/ExternalHistogramStatsMeta.java‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,13 @@ public class ExternalHistogramStatsMeta implements Writable {
4141
@SerializedName("properties")
4242
private Map<String, String> properties;
4343

44+
// The table UUID resolved on the leader at analyze time. It is the key used by the connector
45+
// statistics cache, so persisting it lets followers invalidate the cache during journal replay
46+
// without resolving external table metadata (which may block on HMS/object storage).
47+
// May be null for journals written before this field was introduced (see replay fallback).
48+
@SerializedName("tableUUID")
49+
private String tableUUID;
50+
4451
public ExternalHistogramStatsMeta(String catalogName, String dbName, String tableName, String column,
4552
StatsConstants.AnalyzeType type,
4653
LocalDateTime updateTime,
@@ -70,6 +77,14 @@ public String getColumn() {
7077
return column;
7178
}
7279

80+
public String getTableUUID() {
81+
return tableUUID;
82+
}
83+
84+
public void setTableUUID(String tableUUID) {
85+
this.tableUUID = tableUUID;
86+
}
87+
7388
public StatsConstants.AnalyzeType getType() {
7489
return type;
7590
}

0 commit comments

Comments
 (0)