Pipe: improve progress coverage checks#17940
Conversation
| private static final String SHUTDOWN_PROGRESS_NOT_CONFIRMED_MESSAGE = | ||
| "\u672c\u6b21 shutdown progress \u672a\u786e\u8ba4\u843d\u5230 ConfigNode."; |
There was a problem hiding this comment.
Done. Removed the hard-coded Chinese suffix and moved the shutdown progress confirmation message into DataNodePipeMessages for both en/zh locales.
| LOGGER.info( | ||
| "Start to persist all pipe progress indexes during shutdown, pipe count {}, deadline {} ms", | ||
| getPipeCount(), | ||
| normalizedTimeoutInMs); |
There was a problem hiding this comment.
Done. All newly added shutdown progress logs now use DataNodePipeMessages, with English and Chinese i18n entries.
| LOGGER.warn( | ||
| "Collected empty pipe metas for {} pipes during shutdown. " | ||
| + SHUTDOWN_PROGRESS_NOT_CONFIRMED_MESSAGE, | ||
| pipeCount); |
There was a problem hiding this comment.
Done. Changed this log to info. The method still returns false when pipeCount is non-zero but no meta is collected, so the shutdown hook keeps the single final warning that progress was not confirmed.
| } catch (final Exception e) { | ||
| LOGGER.warn(e.getMessage()); | ||
| LOGGER.warn( | ||
| "Exception occurred while persisting all pipe progress indexes during shutdown. " | ||
| + SHUTDOWN_PROGRESS_NOT_CONFIRMED_MESSAGE, | ||
| e); | ||
| return false; |
There was a problem hiding this comment.
May ignore interruption here
There was a problem hiding this comment.
Done. The join interruption now restores the interrupt flag and logs at info without interrupting the daemon persist thread.
2 participants
Description
Progress coverage
Add
ProgressIndex.isEqualOrAfter(...)and use it for pipe progress coverage checks. Historical TsFile selection now checks whether the start index already covers the resource progress by reusingupdateToMinimumEqualOrIsAfterProgressIndex, which handles Hybrid/Recover partial-dimension cases consistently.Diagnostics
Add historical TsFile selection summary counters for progress-uncovered selection, unclosed/closing selection, time/path filtering, covered skips, deleted/generated-by-pipe skips, and pin failures. Shutdown progress persistence now logs pipe/meta counts, meta size, heartbeat timing, and reports clearly when shutdown progress is not confirmed by ConfigNode before the deadline.
Tests
Add Hybrid/Recover coverage cases for historical TsFile selection and realtime TsFile epoch progress checks.
Validation:
.\mvnw.cmd -Ddevelocity.off=true -pl iotdb-core/node-commons,iotdb-core/datanode spotless:applygit diff --check origin/masterTargeted tests were attempted locally, but DataNode test compilation is currently blocked by unrelated existing errors in generated query/parser and queue classes.
This PR has:
Key changed/added classes (or packages if there are too many classes) in this PR
ProgressIndexPipeHistoricalDataRegionTsFileAndDeletionSourcePipeTsFileEpochProgressIndexKeeperPipeDataNodeTaskAgentDataNodeShutdownHook