fix(ci-dashboard): refine external dependency classification (#461)

dillon-zheng · web-flow · commit a6551837db6b · 2026-05-22T16:55:39.000+08:00
## Summary
- prefer explicit realtikv duplicate-key evidence over earlier Bazel
fetch noise in realcluster and ghpr_check2 logs
- classify Bazel remote cache 429/SlowDown as `INFRA/NETWORK` instead of
`INFRA/EXTERNAL_DEP`
- add rule-engine coverage for the mixed-log regression cases

## Verification
- `uv run --with pytest --python 3.11 python -m pytest
tests/jobs/test_rule_engine.py -k 'realcluster or remote_cache or
ghpr_check2 or external_dependency'`
diff --git a/ci-dashboard/src/ci_dashboard/jobs/error_classification_guidance.py b/ci-dashboard/src/ci_dashboard/jobs/error_classification_guidance.py
@@ -7,11 +7,14 @@
 - Stage names or wrapper steps such as "prepare depends", "build", or generic pipeline/config steps are weak hints; classify the underlying compiler, dependency, test, or merge evidence instead.
 - In this autoscaled CI environment, FailedScheduling/Unschedulable/Insufficient resources/taints/PVC waits during pod Pending are normal scheduling pressure, not an error by themselves.
 - Only classify Kubernetes as INFRA when there is final pod/container failure evidence such as TerminationByKubelet, OOMKilled, or Evicted.
+- Do not classify K8S_MEMORY_EVICTION from partial memory-usage snippets alone; require final kubelet pod-failure evidence such as `Pod [Failed][TerminationByKubelet] The node was low on resource: memory`.
 - Container termination reason OOMKilled is INFRA/OOMKILLED, even when Jenkins later reports a matrix branch failure or agent disconnect.
 - Use job names as strong hints for test families: ghpr/pull unit/mysql jobs are UT, integration/realcluster/lightning/br/dm integration jobs are IT.
 - ghpr_check2 test-failure evidence should usually stay in the UT family because that job behaves like a unit-style mixed check job; do not move it to IT unless the failing evidence is explicitly integration-specific.
 - ghpr_check2 timeout evidence can move to IT/TIMEOUT when the failing branch is explicitly integration-specific, for example `run_real_tikv_tests.sh ...` or `integrationtest_with_tikv.sh ...`, even though the outer job name is ghpr_check2.
 - ghpr_check2 realtikv/integration failure evidence such as `//tests/realtikvtest/... FAILED`, `Test output for //tests/realtikvtest/...`, or PD `NOT_BOOTSTRAPPED` errors inside `run_real_tikv_tests.sh` or `integrationtest_with_tikv.sh` branches should be IT/TEST_FAILURE, not UT/TEST_FAILURE.
+- pull/release `pull_check2` jobs with explicit `realtikv` target failures such as `//tests/realtikvtest/... FAILED`, `Test output for //tests/realtikvtest/...`, or PD `NOT_BOOTSTRAPPED` errors should also be IT/TEST_FAILURE rather than OTHERS/UNCLASSIFIED.
+- In realcluster or ghpr_check2 mixed logs, explicit add-index/import-into duplicate-key evidence such as `[kv:1062]Duplicate entry`, `ErrFoundDuplicateKey`, or `run add index job failed, convert job to rollback` should be IT/TEST_FAILURE even if the same console earlier contains Bazel dependency download failures.
 - Go compiler errors such as "undefined:" or "has no field or method" are BUILD/COMPILE, even when they appear inside Bazel or wrapper output.
 - Bazel/unit summaries such as "FAILED TO BUILD", "fails to build", or "0 failing out of 0 test cases" point to an upstream build problem, not a UT test failure by themselves.
 - Failpoint rewrite/parser errors such as "Rewrite error ... expected declaration/statement, found '<<'" are BUILD/COMPILE.
@@ -22,14 +25,24 @@
 - Support-repo checkout failures that log `computeBranchFromPR component: tidb-test`, warn that the `tidb-test` PR base branch must match the `tidb` PR base branch, and then fail with `fatal: couldn't find remote ref refs/heads/...` are BUILD/PIPELINE_CONFIG; treat them as CI checkout/refspec logic failures rather than INFRA/GIT noise.
 - Direct main-repo checkout failures such as `fatal: couldn't find remote ref refs/pull/<num>/head` are INFRA/GIT, not BUILD/PIPELINE_CONFIG; they often reflect a transient GitHub pull-ref visibility issue, and later matrix or agent noise should be ignored.
 - Nogo or build-time static-analysis validation failures such as "Validating nogo output" or "Running nogo on //... failed" are BUILD/FORMAT_CHECK, including ghpr_build, ghpr_check2, pull_build_next_gen, and pull_unit_test jobs; do not leave them in BUILD/PIPELINE_CONFIG or move them to BUILD/COMPILE.
+- This also applies when those `nogo` / `Validating nogo output` failures happen inside integration-family wrappers such as `pull_integration_realcluster_test_next_gen`; they are still BUILD/FORMAT_CHECK.
 - Repository/archive download failures such as "Error downloading", "Error computing the main repository mapping", download 404/429/502/503/504, or read-timeout while fetching Bazel/GitHub dependencies are INFRA/EXTERNAL_DEP even if matrix branches later show test interruptions.
+- Bazel remote-cache throttling evidence such as `Remote Cache: 429 Too Many Requests`, `Failed to fetch blobs because of a remote cache error`, or GCS `SlowDown ... exceeded your bucket's IO capacity` is INFRA/NETWORK rather than INFRA/EXTERNAL_DEP.
+- Bazel server failures such as `Server terminated abruptly (error code: 14, error message: 'Socket closed')` are INFRA/NETWORK when there is no earlier stronger build/test root cause.
 - For integration job families, Jenkins/test wrapper timeout evidence such as "Timeout has been exceeded" is IT/TIMEOUT, not INFRA/NETWORK.
 - BR integration matrix TEST_GROUP failures are IT/TEST_FAILURE, even if the failed case logs local 127.0.0.1 PD/TiKV connection-refused or timeout symptoms.
+- merged integration matrix failures should usually stay in IT/TEST_FAILURE when the failed branch also shows concrete product/test evidence such as `TEST FAILED`, duplicate-key exceptions, JDBC assertion failures, BR store-backup failures, or local service connection-refused inside the test workload.
+- `merged_integration_mysql_test` query/result mismatches such as `query succeeded, but expected error(s)!` or `query failed with non expected error(s)!` are IT/TEST_FAILURE.
 - BR restore compatibility failures such as "ErrRestoreNotFreshCluster", "ErrRestoreIncompatibleSys", "missing column in cluster data", or "incompatible column, table:" are IT/TEST_FAILURE, even if later console tail shows scatter warnings or Jenkins remoting noise.
+- For `pull_cdc_storage_integration_light`, matrix `TEST_GROUP` failures with `TEST FAILED`, `check diff failed`, `check data failed`, or `run task failed` are IT/TEST_FAILURE, even if the same build also had transient Pending/PVC scheduling noise.
+- For `pull_cdc_storage_integration_heavy_next_gen`, matrix/data-check failures such as `check diff failed`, `ErrTableIsNotFounded`, or `ErrMaintainerNotFounded` are IT/TEST_FAILURE.
+- For `ghpr_check`, `make ... integrationtest` failures with plan diff or integration output mismatch evidence are IT/TEST_FAILURE rather than INFRA/NETWORK.
 - Kubelet final pod failures such as `TerminationByKubelet` with `imminent node shutdown` are INFRA/K8S and should beat later Jenkins remoting or agent-offline noise.
 - Jenkins Groovy/runtime/cache/websocket/controller persistence errors are INFRA subcategories, not product BUILD failures.
+- If a log first shows checkout interruption or agent-loss evidence such as `ERROR: Checkout failed`, `java.lang.InterruptedException`, or `AgentOfflineException: Unable to create live FilePath`, and only later shows `groovy.lang.MissingPropertyException: No such property: source`, prefer INFRA/JENKINS_AGENT_OFFLINE over INFRA/JENKINS_GROOVY.
 - Agent disconnect/remoting failures such as `Timeout waiting for agent to come back`, `AgentOfflineException`, or `was marked offline: Connection was broken` are INFRA/JENKINS_AGENT_OFFLINE when there is no earlier stronger root cause.
 - Disk-full evidence such as "No space left on device" is INFRA/DISK_FULL even when it appears while Jenkins saves pipeline state.
+- TiDB ingest environment errors such as `Error 8256 (HY000): Check ingest environment failed: no enough space in /tmp/tidb/...` are also INFRA/DISK_FULL.
 - Prow superseded aborts are OTHERS/SUPERSEDED_BY_NEWER_BUILD when Prow marks the job aborted because a newer same-PR same-job version is running, or when an admin-abort log has same-PR same-job newer different-SHA build evidence; this overrides downstream noise.
 - Admin aborts are OTHERS/ABORT_BY_ADMIN and override downstream matrix, network, or interrupted-process symptoms.
 - Merge conflicts such as "CONFLICT (content)" or "Automatic merge failed" are OTHERS/CODE_CONFLICT, because they are neither infra nor product test/build quality failures.
diff --git a/ci-dashboard/src/ci_dashboard/jobs/error_taxonomy.json b/ci-dashboard/src/ci_dashboard/jobs/error_taxonomy.json
@@ -79,7 +79,8 @@
       "l2": "DISK_FULL",
       "text_patterns": [
         "java\\.io\\.IOException: No space left on device",
-        "No space left on device"
+        "No space left on device",
+        "Error 8256 \\(HY000\\): Check ingest environment failed: no enough space in /tmp/tidb/"
       ]
     },
     {
@@ -98,20 +99,34 @@
       "l2": "K8S_MEMORY_EVICTION",
       "text_patterns": [
         "Pod \\[Failed\\]\\[TerminationByKubelet\\] The node was low on resource: memory",
-        "The node was low on resource: memory\\.[\\s\\S]*Container .* was using .* request is .* larger consumption of memory",
-        "Container .* was using .* request is .* larger consumption of memory"
+        "Pod \\[Failed\\]\\[TerminationByKubelet\\] The node was low on resource: memory\\.[\\s\\S]*Container .* was using .* request is .* larger consumption of memory"
       ]
     },
     {
       "name": "realcluster_statistics_duplicate_key_test_failure",
       "l1": "IT",
       "l2": "TEST_FAILURE",
       "job_name_patterns": [
+        "(^|/)ghpr_check2$",
+        "(^|_)ghpr_check2($|_)",
         "(^|/)pull_integration_realcluster_test(_next_gen)?$",
         "(^|_)pull_integration_realcluster_test(_next_gen)?($|_)"
       ],
       "text_patterns": [
-        "Failed in branch Matrix - SCRIPT_AND_ARGS = 'tests/realtikvtest/scripts/next-gen/run-tests\\.sh bazel_statisticstest'[\\s\\S]*Duplicate entry [^\\r\\n]+ for key '(?:t\\.idx|t0\\.idx1)'"
+        "Failed in branch Matrix - SCRIPT_AND_ARGS = 'tests/realtikvtest/scripts/next-gen/run-tests\\.sh bazel_statisticstest'[\\s\\S]*Duplicate entry [^\\r\\n]+ for key '(?:t\\.idx|t0\\.idx1)'",
+        "Failed in branch Matrix - SCRIPT_AND_ARGS = '(?:tests/realtikvtest/scripts/next-gen/run-tests\\.sh|run_real_tikv_tests\\.sh) bazel_(?:addindextest[0-9]*|importintotest[0-9]*)'[\\s\\S]*(?:\\[kv:1062\\]Duplicate entry|ErrFoundDuplicateKey|run add index job failed, convert job to rollback)",
+        "Test output for //tests/realtikvtest/[^\\r\\n]+[\\s\\S]*(?:\\[kv:1062\\]Duplicate entry|ErrFoundDuplicateKey|run add index job failed, convert job to rollback)",
+        "//tests/realtikvtest/[^\\r\\n]+ FAILED in [\\s\\S]*(?:\\[kv:1062\\]Duplicate entry|ErrFoundDuplicateKey|run add index job failed, convert job to rollback)"
+      ]
+    },
+    {
+      "name": "infra_remote_cache_throttle_network",
+      "l1": "INFRA",
+      "l2": "NETWORK",
+      "text_patterns": [
+        "Remote Cache: 429 Too Many Requests",
+        "Failed to fetch blobs because of a remote cache error\\.: 429 Too Many Requests",
+        "SlowDown[^\\r\\n]*exceeded your bucket's IO capacity"
       ]
     },
     {
@@ -152,11 +167,13 @@
         "(^|/)ghpr_check2$",
         "(^|/)ghpr_build$",
         "(^|/)pull_build_next_gen$",
+        "(^|/)pull_integration_realcluster_test(_next_gen)?$",
         "(^|/)ghpr_unit_test$",
         "(^|/)pull_unit_test[^/]*$",
         "(^|_)ghpr_check2($|_)",
         "(^|_)ghpr_build($|_)",
         "(^|_)pull_build_next_gen($|_)",
+        "(^|_)pull_integration_realcluster_test(_next_gen)?($|_)",
         "(^|_)ghpr_unit_test($|_)",
         "(^|_)pull_unit_test($|_)"
       ],
@@ -171,7 +188,9 @@
       "l2": "TEST_FAILURE",
       "job_name_patterns": [
         "(^|/)ghpr_check2$",
-        "(^|_)ghpr_check2($|_)"
+        "(^|/)pull_check2[^/]*$",
+        "(^|_)ghpr_check2($|_)",
+        "(^|_)pull_check2($|_)"
       ],
       "text_patterns": [
         "//tests/realtikvtest/[^\\r\\n]+ FAILED in ",
@@ -193,9 +212,12 @@
       ],
       "text_patterns": [
         "Testing //[^\\r\\n]+ failed: Test failed",
+        "FAIL: //[^\\r\\n]+_test",
         "Test cases: finished with [^\\r\\n]*[1-9][0-9]* failing",
         "Build completed, [1-9][0-9]* test FAILED",
         "Executed [0-9]+ out of [0-9]+ test: [^\\r\\n]*fails locally",
+        "testdata: Errors on loading test data from file:",
+        "panic: runtime error: invalid memory address or nil pointer dereference",
         "//tests/realtikvtest/[^\\r\\n]+ FAILED in ",
         "--- FAIL: "
       ]
@@ -206,7 +228,9 @@
       "l2": "TIMEOUT",
       "job_name_patterns": [
         "(^|/)ghpr_check2$",
-        "(^|_)ghpr_check2($|_)"
+        "(^|/)pull_check2[^/]*$",
+        "(^|_)ghpr_check2($|_)",
+        "(^|_)pull_check2($|_)"
       ],
       "text_patterns": [
         "Timeout has been exceeded[\\s\\S]*(run_real_tikv_tests\\.sh|integrationtest_with_tikv\\.sh)",
@@ -332,6 +356,38 @@
         "run task failed [1-9][0-9]*-th time"
       ]
     },
+    {
+      "name": "cdc_storage_integration_heavy_next_gen_test_failure",
+      "l1": "IT",
+      "l2": "TEST_FAILURE",
+      "job_name_patterns": [
+        "(^|/)pull_cdc_storage_integration_heavy_next_gen$",
+        "(^|_)pull_cdc_storage_integration_heavy_next_gen($|_)"
+      ],
+      "text_patterns": [
+        "Failed in branch Matrix - TEST_GROUP = '[^']+'",
+        "check diff failed [1-9][0-9]*-th time",
+        "\\[CDC:ErrTableIsNotFounded\\]table is not found",
+        "\\[CDC:ErrMaintainerNotFounded\\]maintainer is not found"
+      ]
+    },
+    {
+      "name": "cdc_storage_integration_light_test_failure",
+      "l1": "IT",
+      "l2": "TEST_FAILURE",
+      "job_name_patterns": [
+        "(^|/)pull_cdc_storage_integration_light$",
+        "(^|_)pull_cdc_storage_integration_light($|_)"
+      ],
+      "text_patterns": [
+        "Failed in branch Matrix - TEST_GROUP = '[^']+'",
+        "TEST FAILED: OUTPUT DOES NOT CONTAIN",
+        "TEST FAILED: OUTPUT CONTAINS",
+        "check data failed [1-9][0-9]*-th time",
+        "check diff failed [1-9][0-9]*-th time",
+        "run task failed [1-9][0-9]*-th time"
+      ]
+    },
     {
       "name": "cdc_integration_kafka_test_failure",
       "l1": "IT",
@@ -360,6 +416,49 @@
         "make: \\*\\*\\* \\[Makefile:[0-9]+: push-down-test\\] Error [0-9]+"
       ]
     },
+    {
+      "name": "merged_integration_mysql_test_failure",
+      "l1": "IT",
+      "l2": "TEST_FAILURE",
+      "job_name_patterns": [
+        "(^|/)merged_integration_mysql_test$",
+        "(^|_)merged_integration_mysql_test($|_)"
+      ],
+      "text_patterns": [
+        "query succeeded, but expected error\\(s\\)!",
+        "query failed with non expected error\\(s\\)!",
+        "Failed in branch Matrix - TEST_PART = '[^']+'"
+      ]
+    },
+    {
+      "name": "merged_unit_test_failure",
+      "l1": "UT",
+      "l2": "TEST_FAILURE",
+      "job_name_patterns": [
+        "(^|/)merged_unit_test$",
+        "(^|_)merged_unit_test($|_)"
+      ],
+      "text_patterns": [
+        "=== FAIL:",
+        "--- FAIL:",
+        "Received unexpected error: Get \\\"http://127\\.0\\.0\\.1:[0-9]+/metrics\\\":"
+      ]
+    },
+    {
+      "name": "merged_integration_matrix_test_failure",
+      "l1": "IT",
+      "l2": "TEST_FAILURE",
+      "job_name_patterns": [
+        "(^|/)merged_integration_.*_test$",
+        "(^|_)merged_integration_.*_test($|_)"
+      ],
+      "text_patterns": [
+        "Failed in branch Matrix - [^\\r\\n]+[\\s\\S]*TEST FAILED: OUTPUT (?:DOES NOT CONTAIN|CONTAINS)",
+        "Failed in branch Matrix - [^\\r\\n]+[\\s\\S]*org\\.jooq\\.exception\\.DataAccessException: SQL \\[[^\\r\\n]+Duplicate entry",
+        "Failed in branch Matrix - [^\\r\\n]+[\\s\\S]*SQLIntegrityConstraintViolationException: Duplicate entry",
+        "Failed in branch Matrix - [^\\r\\n]+[\\s\\S]*(?:retryable storage error|store backup failed|\\[BR:Common:ErrFailedToConnect\\] failed to make connection to store|\\[health check\\] check health error)"
+      ]
+    },
     {
       "name": "merged_integration_cleanup_pipeline_config_failure",
       "l1": "BUILD",
@@ -409,23 +508,42 @@
         "\\.go:[0-9]+:[0-9]+: .* undefined( \\(type .*\\))?",
         "\\.go:[0-9]+:[0-9]+: cannot use .* as .*",
         "\\.go:[0-9]+:[0-9]+: .* does not implement .*",
+        "\\.go:[0-9]+:[0-9]+: unknown field ",
         "\\.go:[0-9]+:[0-9]+: too many arguments in call to",
         "\\.go:[0-9]+:[0-9]+: not enough arguments in call to",
         "Rewrite error [^\\r\\n]+\\.go:[0-9]+:[0-9]+: expected (statement|declaration), found ",
         "plugin\\.Open\\([^\\r\\n]+\\): plugin was built with a different version of package "
       ]
     },
+    {
+      "name": "ghpr_check_integrationtest_failure",
+      "l1": "IT",
+      "l2": "TEST_FAILURE",
+      "job_name_patterns": [
+        "(^|/)ghpr_check$",
+        "(^|_)ghpr_check($|_)"
+      ],
+      "text_patterns": [
+        "diff:[\\s\\S]*make: \\*\\*\\* \\[Makefile:[0-9]+: integrationtest\\] Error [0-9]+",
+        "make: \\*\\*\\* \\[Makefile:[0-9]+: integrationtest\\] Error [0-9]+"
+      ]
+    },
     {
       "name": "build_codegen_failure",
       "l1": "BUILD",
       "l2": "CODEGEN",
       "job_name_patterns": [
         "(^|/)ghpr_check$",
+        "(^|/)pull_check[^/]*$",
         "(^|_)ghpr_check($|_)"
+        ,
+        "(^|_)pull_check($|_)"
       ],
       "text_patterns": [
         "Your commit is changed after running go generate \\./\\.\\.\\., it should not happen\\.",
-        "make: \\*\\*\\* \\[Makefile:[0-9]+: gogenerate\\] Error [0-9]+"
+        "make: \\*\\*\\* \\[Makefile:[0-9]+: gogenerate\\] Error [0-9]+",
+        "Files go\\.sum and /tmp/go\\.sum\\.before differ",
+        "make: \\*\\*\\* \\[Makefile:[0-9]+: tidy\\] Error [0-9]+"
       ]
     },
     {
@@ -453,7 +571,9 @@
         "hudson\\.FilePath\\.readFromTar\\(FilePath\\.java:[0-9]+\\)[\\s\\S]*Caused: java\\.io\\.IOException: Failed to extract input stream",
         "Caused: java\\.io\\.IOException: Failed to extract input stream[\\s\\S]*hudson\\.FilePath\\.readFromTar\\(FilePath\\.java:[0-9]+\\)",
         "Caused: java\\.io\\.IOException: Failed to extract input stream[\\s\\S]*jenkins-pipeline-cache",
-        "org\\.apache\\.http\\.ConnectionClosedException: Premature end of Content-Length delimited message body[\\s\\S]*jenkins-pipeline-cache"
+        "org\\.apache\\.http\\.ConnectionClosedException: Premature end of Content-Length delimited message body[\\s\\S]*jenkins-pipeline-cache",
+        "Workspace has a \\.git repository, but it appears to be corrupt\\.",
+        "fatal: not a gitdir "
       ]
     },
     {
@@ -465,7 +585,8 @@
         "okhttp3\\.internal\\.ws\\.WebSocketReader",
         "okhttp3\\.internal\\.ws\\.RealWebSocket",
         "java\\.io\\.EOFException[\\s\\S]*okhttp3\\.internal\\.ws",
-        "ERROR: Process exited immediately after creation\\.[\\s\\S]*websocket"
+        "ERROR: Process exited immediately after creation\\.[\\s\\S]*websocket",
+        "Server terminated abruptly \\(error code: 14, error message: 'Socket closed'\\)"
       ]
     },
     {
@@ -481,6 +602,19 @@
         "ERROR: internal error performing authentication[\\s\\S]*fatal: Could not read from remote repository\\."
       ]
     },
+    {
+      "name": "infra_jenkins_agent_offline_before_groovy_missing_property",
+      "l1": "INFRA",
+      "l2": "JENKINS_AGENT_OFFLINE",
+      "job_name_patterns": [
+        "(^|/)merged_sqllogic_test$",
+        "(^|_)merged_sqllogic_test($|_)"
+      ],
+      "text_patterns": [
+        "AgentOfflineException: Unable to create live FilePath[\\s\\S]*groovy\\.lang\\.MissingPropertyException: No such property: source",
+        "ERROR: Checkout failed[\\s\\S]*java\\.lang\\.InterruptedException[\\s\\S]*groovy\\.lang\\.MissingPropertyException: No such property: source"
+      ]
+    },
     {
       "name": "infra_jenkins_groovy",
       "l1": "INFRA",
@@ -549,7 +683,6 @@
       "l2": "EXTERNAL_DEP",
       "text_patterns": [
         "rate limit exceeded",
-        "429 Too Many Requests",
         "502 Bad Gateway",
         "503 Service Unavailable",
         "upstream connect error"
diff --git a/ci-dashboard/tests/jobs/test_rule_engine.py b/ci-dashboard/tests/jobs/test_rule_engine.py
