Skip to content

refactor: migrate SQL parser from arana-db to pingcap/tidb#1120

Open
yunfan24 wants to merge 15 commits into
apache:masterfrom
yunfan24:sql-praser
Open

refactor: migrate SQL parser from arana-db to pingcap/tidb#1120
yunfan24 wants to merge 15 commits into
apache:masterfrom
yunfan24:sql-praser

Conversation

@yunfan24

@yunfan24 yunfan24 commented May 9, 2026

Copy link
Copy Markdown
Member
  • I have registered the PR changes.

What this PR does:

  • Migrate SQL parser dependency from github.com/arana-db/parser to github.com/pingcap/tidb/pkg/parser
  • Adapt API changes: PatternLikeExprPatternLikeOrIlikeExpr
  • Add assignParamMarkerOrders to handle parameter marker ordering required by the new parser
  • Run go mod tidy to resolve transitive dependencies
    Which issue(s) this PR fixes:

#1104

Special notes for your reviewer:

  • go.mod version bump (1.20 → 1.25)

Does this PR introduce a user-facing change?:


@github-actions github-actions Bot added the ci/cd label May 12, 2026
@github-actions github-actions Bot added the rm label May 13, 2026
@codecov-commenter

codecov-commenter commented May 14, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 40.72165% with 115 lines in your changes missing coverage. Please review.
✅ Project coverage is 54.79%. Comparing base (84c991f) to head (213a058).
⚠️ Report is 3 commits behind head on master.

Files with missing lines Patch % Lines
pkg/datasource/sql/xa_resource_manager.go 0.00% 17 Missing ⚠️
...ne/engine/core/process_ctrl_statemachine_engine.go 0.00% 14 Missing ⚠️
pkg/datasource/sql/xa/oracle_xa_connection.go 0.00% 7 Missing ⚠️
pkg/datasource/sql/exec/at/insert_executor.go 20.00% 2 Missing and 2 partials ⚠️
.../sql/undo/builder/mysql_insert_undo_log_builder.go 20.00% 2 Missing and 2 partials ⚠️
...g/saga/statemachine/engine/invoker/grpc_invoker.go 33.33% 4 Missing ⚠️
pkg/saga/statemachine/store/db/db.go 0.00% 2 Missing and 2 partials ⚠️
pkg/datasource/sql/exec/at/base_executor.go 76.92% 3 Missing ⚠️
...tasource/sql/exec/at/select_for_update_executor.go 25.00% 3 Missing ⚠️
pkg/rm/tcc/tcc_resource.go 0.00% 3 Missing ⚠️
... and 35 more
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1120      +/-   ##
==========================================
- Coverage   58.36%   54.79%   -3.58%     
==========================================
  Files         282      308      +26     
  Lines       19405    20670    +1265     
==========================================
  Hits        11326    11326              
- Misses       7102     8366    +1264     
- Partials      977      978       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@yunfan24 yunfan24 marked this pull request as ready for review May 18, 2026 18:47
@thunguo

thunguo commented May 23, 2026

Copy link
Copy Markdown
Contributor

Resolve the CI problems and add some unit-test cases.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR migrates Seata-Go’s MySQL SQL parsing layer (used by AT-mode executors / undo-log builders) from github.com/arana-db/parser to the upstream TiDB parser (github.com/pingcap/tidb/pkg/parser), and updates call sites/tests accordingly. It also includes a broad Go toolchain / dependency update and various small refactors.

Changes:

  • Replace Arana parser imports with TiDB parser imports across SQL parsing, executors, and undo-log builders; adapt AST API differences (e.g., PatternLikeExprPatternLikeOrIlikeExpr).
  • Add parameter-marker ordering assignment (assignParamMarkerOrders) to support TiDB parser’s param marker order handling.
  • Update Go module/tooling and CI configs (Go version bump, golangci-lint config schema, dependency tidy) plus minor cleanups (error strings, ioutil removal, etc.).

Reviewed changes

Copilot reviewed 92 out of 93 changed files in this pull request and generated 11 comments.

Show a summary per file
File Description
pkg/util/reflectx/unmarkshaler.go Minor reflection validation refactor in setter signature check.
pkg/util/reflectx/unexpoert_field.go Pointer-kind handling updated in GetElemDataValue.
pkg/util/reflectx/reference_service.go Pointer-kind handling updated for service reference derivation.
pkg/util/net/address_validator.go Regex literals simplified using raw string literals.
pkg/util/convert/convert.go Reflection-based assignment pointer-kind handling updated.
pkg/tm/transaction_executor.go Error message punctuation normalized (tests updated accordingly).
pkg/tm/transaction_executor_test.go Update expected error messages to match new strings.
pkg/tm/global_transaction.go Error message punctuation normalized.
pkg/tm/global_transaction_test.go Update expected error strings to match new strings.
pkg/saga/statemachine/store/db/statelog.go Simplify nil/len check for serialized error bytes.
pkg/saga/statemachine/store/db/db.go Add rows.Err() checks to SelectOne/SelectList.
pkg/saga/statemachine/statelang/parser/sub_state_machine_parser.go Parser method call cleanups (GetString, task attr parsing).
pkg/saga/statemachine/statelang/parser/statemachine_parser.go Simplify make(map...) initialization.
pkg/saga/statemachine/statelang/parser/statemachine_json_parser.go Simplify boolean return helper.
pkg/saga/statemachine/statelang/parser/statemachine_config_parser.go Simplify empty content checks for JSON/YAML config parsers.
pkg/saga/statemachine/process_ctrl/process_context.go Simplify map lookups / deletes.
pkg/saga/statemachine/process_ctrl/handlers/service_task_state_handler.go Simplify empty slice check for sub-instances.
pkg/saga/statemachine/process_ctrl/event_bus.go Log formatting adjusted to avoid treating dynamic strings as format strings.
pkg/saga/statemachine/engine/strategy/status_decision.go Simplify slice length check.
pkg/saga/statemachine/engine/pcext/state_router_impl.go Use promoted AbstractTaskState on embedded service task impl.
pkg/saga/statemachine/engine/pcext/process_router.go Simplify map nil/len check.
pkg/saga/statemachine/engine/pcext/process_handler.go Remove redundant nil checks; simplify expression resolver variable.
pkg/saga/statemachine/engine/pcext/loop_task_utils.go Use promoted AbstractTaskState on embedded service task impl.
pkg/saga/statemachine/engine/pcext/instruction.go Replace errors.New(fmt.Sprintf(...)) with fmt.Errorf.
pkg/saga/statemachine/engine/invoker/local_invoker.go Pointer-type detection updated when converting params.
pkg/saga/statemachine/engine/invoker/grpc_invoker.go Replace errors.New(fmt.Sprintf(...)) with fmt.Errorf.
pkg/saga/statemachine/engine/invoker/grpc_invoker_test.go Simplify slice empty checks in tests.
pkg/saga/statemachine/engine/invoker/func_invoker_test.go Simplify slice empty checks in tests.
pkg/saga/statemachine/engine/exception/forward_invalid_exception_test.go Update exception message field access in tests.
pkg/saga/statemachine/engine/exception/exception_test.go Update exception message field access in tests.
pkg/saga/statemachine/engine/core/process_ctrl_statemachine_engine.go Simplify nil checks; log call simplification; map iteration simplification.
pkg/rm/tcc/tcc_service.go Use proxy methods for action metadata; map/type switch simplification; pointer-kind handling updated.
pkg/rm/tcc/tcc_resource.go Use promoted proxy methods; call commit/rollback through resource wrapper.
pkg/rm/tcc/fence/store/db/dao/tcc_fence_db.go Add rows.Err() check after scanning query results.
pkg/rm/rm_remoting.go Normalize error message casing.
pkg/remoting/getty/session_manager.go Pointer-kind handling updated for reflection-based XID access.
pkg/remoting/getty/readwriter.go Refactor heartbeat message type handling to switch.
pkg/integration/grpc/grpc_transaction_interceptor.go Simplify metadata slice length checks.
pkg/discovery/etcd3_test.go Simplify logging call.
pkg/datasource/sql/xa/oracle_xa_connection.go Normalize “oracle” error string casing.
pkg/datasource/sql/xa/mysql_xa_connection.go Remove redundant break; simplify infinite loop syntax.
pkg/datasource/sql/xa_resource_manager.go Refactor ticker loop; simplify error logging formatting.
pkg/datasource/sql/util/lockkey.go Use fmt.Fprintf instead of fmt.Sprintf for builder writes.
pkg/datasource/sql/util/escape.go Use strings.Contains instead of strings.Index.
pkg/datasource/sql/util/convert.go Reflection-based assignment pointer-kind handling updated.
pkg/datasource/sql/undo/executor/utils.go Use fmt.Fprintf for builder writes when emitting PK column names.
pkg/datasource/sql/undo/builder/mysql_update_undo_log_builder.go Migrate parser imports to TiDB; simplify before-image checks.
pkg/datasource/sql/undo/builder/mysql_update_undo_log_builder_test.go Update parser test driver import to TiDB.
pkg/datasource/sql/undo/builder/mysql_multi_update_undo_log_builder.go Migrate parser imports to TiDB.
pkg/datasource/sql/undo/builder/mysql_multi_update_undo_log_builder_test.go Update parser imports/test driver to TiDB.
pkg/datasource/sql/undo/builder/mysql_multi_undo_log_builder.go Remove redundant break statements in switches.
pkg/datasource/sql/undo/builder/mysql_multi_undo_log_builder_test.go Migrate AST/model imports to TiDB.
pkg/datasource/sql/undo/builder/mysql_multi_delete_undo_log_builder.go Migrate parser imports to TiDB.
pkg/datasource/sql/undo/builder/mysql_insertonduplicate_update_undo_log_builder.go Migrate AST imports to TiDB.
pkg/datasource/sql/undo/builder/mysql_insert_undo_log_builder.go Migrate AST imports; normalize error messages; simplify nil/len checks.
pkg/datasource/sql/undo/builder/mysql_insert_undo_log_builder_test.go Migrate parser imports/test driver to TiDB.
pkg/datasource/sql/undo/builder/mysql_delete_undo_log_builder.go Migrate parser imports to TiDB.
pkg/datasource/sql/undo/builder/basic_undo_log_builder.go Migrate parser/test driver; refactor AST traversal; use fmt.Fprintf for builders.
pkg/datasource/sql/undo/base/undo.go Switch parser mysql import to TiDB; add rows.Err() check; simplify map lookup.
pkg/datasource/sql/types/image.go Pointer-kind handling updated for actual value extraction.
pkg/datasource/sql/types/executor.go Migrate AST/format imports to TiDB.
pkg/datasource/sql/types/executor_test.go Migrate AST imports to TiDB.
pkg/datasource/sql/parser/parser_factory.go Switch to TiDB parser; add param marker order assignment visitor.
pkg/datasource/sql/parser/parse_factory_test.go Migrate parser imports/test driver to TiDB.
pkg/datasource/sql/exec/select_for_update_executor.go Migrate parser imports; normalize error strings; use fmt.Fprintf.
pkg/datasource/sql/exec/at/update_join_executor.go Migrate parser imports; normalize error strings.
pkg/datasource/sql/exec/at/update_executor.go Migrate parser imports; normalize error strings.
pkg/datasource/sql/exec/at/select_for_update_executor.go Migrate parser imports; rename conflict error var; normalize errors; use fmt.Fprintf.
pkg/datasource/sql/exec/at/select_for_update_executor_test.go Adjust assertion expression.
pkg/datasource/sql/exec/at/multi_update_excutor.go Migrate parser imports to TiDB.
pkg/datasource/sql/exec/at/multi_delete_executor.go Migrate parser imports; normalize error strings; minor string conversion change.
pkg/datasource/sql/exec/at/insert_on_update_executor.go Migrate AST imports to TiDB.
pkg/datasource/sql/exec/at/insert_executor.go Migrate AST imports; normalize error strings; simplify nil/len checks.
pkg/datasource/sql/exec/at/insert_executor_test.go Migrate parser imports/test driver to TiDB.
pkg/datasource/sql/exec/at/delete_executor.go Migrate parser imports to TiDB.
pkg/datasource/sql/exec/at/base_executor.go Migrate parser/model/test driver; adapt LIKE AST node type; refactor traversal logic; use fmt.Fprintf.
pkg/datasource/sql/driver.go Fix error message spelling (“unsupported”).
pkg/datasource/sql/datasource/utils.go Pointer-kind handling updated for DeepEqual.
pkg/datasource/sql/datasource/mysql/trigger.go Minor comparison style change.
pkg/datasource/sql/conn_xa.go Use c.targetConn directly when creating XA resource.
pkg/datasource/sql/conn_at.go Simplify ExecResult handling in query wrapper logic.
pkg/compressor/gzip_compress.go Replace deprecated ioutil.ReadAll with io.ReadAll.
pkg/compressor/deflate_compress_test.go Simplify variable declaration.
pkg/compressor/bzip2_compress.go Replace deprecated ioutil.ReadAll with io.ReadAll.
pkg/client/config.go Replace deprecated ioutil.ReadFile with os.ReadFile.
go.sum Update checksums to reflect dependency migration and tidy.
go.mod Replace parser dependency; bump Go version; update various direct/indirect deps.
.golangci.yml Migrate golangci-lint config to v2 schema; adjust enabled linters/formatters.
.github/workflows/unit-test.yml Update Go version and GitHub Actions versions.
.github/workflows/license.yml Update Go version used by license check job.
.github/workflows/integrate-test.yml Update Go version used by integration tests.
.github/workflows/golangci-lint.yml Update Go version and golangci-lint action/version.
.github/workflows/build.yml Update Go version used by build job.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread pkg/util/reflectx/unexpoert_field.go
Comment thread pkg/util/reflectx/reference_service.go
Comment thread pkg/util/convert/convert.go
Comment thread pkg/util/convert/convert.go
Comment thread pkg/datasource/sql/util/convert.go
Comment thread pkg/saga/statemachine/engine/invoker/local_invoker.go
Comment thread pkg/datasource/sql/datasource/utils.go
Comment thread pkg/datasource/sql/types/image.go
Comment thread pkg/remoting/getty/session_manager.go
Comment thread pkg/rm/tcc/tcc_service.go
@Ethan-Xingyue

Copy link
Copy Markdown
Contributor

https://github.com/pingcap/tidb/releases/tag/v8.5.6

为了项目长久发展,去当前连链接中去把源码down一下~,不要用go.mod的形式

需要在license和notice里面标注一下项目出处,参考seata-java

@github-actions

Copy link
Copy Markdown

Bot detected the issue body's language is not English, translate it automatically.

https://github.com/pingcap/tidb/releases/tag/v8.5.6

For the long-term development of the project, go to the current link to down the source code ~, do not use the form go.mod

Need to mark the project provenance in the license and notice, refer to seata-java

yunfan24 added 2 commits June 6, 2026 12:25
…aser

# Conflicts:
#	go.mod
#	pkg/datasource/sql/exec/at/base_executor.go
#	pkg/datasource/sql/exec/at/insert_executor_test.go
#	pkg/datasource/sql/undo/executor/utils.go
@ssshr-66 ssshr-66 self-requested a review June 6, 2026 12:06
@Ethan-Xingyue

Copy link
Copy Markdown
Contributor

plz fix ci

@yunfan24

Copy link
Copy Markdown
Member Author

Among the 29 merged PRs in arana-db/parser, a large portion are Arana-specific features, such as sharding rule inspection, topology management, multi-tenant syntax, and DRDS partitioning. seata-go has zero references to these features, and they are unrelated to distributed transactions.
Some features are already supported by newer versions of TiDB, such as SHOW REPLICA STATUS, ADMIN CHECK TABLE, and SHOW EXTENDED COLUMNS/FIELDS.
What seata-go actually depends on is entirely standard parser API functionality: SQL parsing, AST traversal, SQL restoration, and parameter markers. It does not depend on any Arana-specific custom functionality.
The only risk point is that arana-db/parser fills ParamMarkerExpr.Order during parsing, while the official TiDB parser does not. Although TiDB defines the SetOrder interface, parser.y never calls it. seata-go depends on this field for parameter binding, and the migration has already adapted the seata-go code to handle this.
So it seems reasonable to switch directly to the latest TiDB parser, without maintaining a local fork.

@yunfan24

Copy link
Copy Markdown
Member Author

@ssshr-66 @thunguo @Ethan-Xingyue ready for review and merge

@thunguo

thunguo commented Jul 4, 2026

Copy link
Copy Markdown
Contributor

解决一下冲突

@github-actions

github-actions Bot commented Jul 4, 2026

Copy link
Copy Markdown

Bot detected the issue body's language is not English, translate it automatically.

Resolve a conflict

@yunfan24

yunfan24 commented Jul 8, 2026

Copy link
Copy Markdown
Member Author

解决一下冲突

@thunguo done

@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown

Bot detected the issue body's language is not English, translate it automatically.

Resolve a conflict

@ thunguo done

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants