Skip to content

[spark] Support V2 UPDATE for data evolution tables#8214

Open
kerwin-zk wants to merge 1 commit into
apache:masterfrom
kerwin-zk:spark-v2-dml-data-evolution
Open

[spark] Support V2 UPDATE for data evolution tables#8214
kerwin-zk wants to merge 1 commit into
apache:masterfrom
kerwin-zk:spark-v2-dml-data-evolution

Conversation

@kerwin-zk

@kerwin-zk kerwin-zk commented Jun 12, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Purpose

Support V2 UPDATE for data evolution tables.

Tests

CI

@JingsongLi

JingsongLi commented Jun 12, 2026

Copy link
Copy Markdown
Contributor

How to support delete row ids? cc @leaves12138

@kerwin-zk

kerwin-zk commented Jun 12, 2026

Copy link
Copy Markdown
Contributor Author

How to support delete row ids? cc @leaves12138

@JingsongLi Deleted row ids are simply retired — they become holes in the row-id space. Surviving rows always keep their original row ids, and no physical _ROW_ID column is ever written, so row ids stay derived from firstRowId + position, exactly like the files produced by DataEvolutionDeleteRewriter in #8182

JingsongLi
JingsongLi reviewed Jun 13, 2026
View reviewed changes

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 from my side, no blocking issues found

@JingsongLi

JingsongLi commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

How to support delete row ids? cc @leaves12138

@JingsongLi Deleted row ids are simply retired — they become holes in the row-id space. Surviving rows always keep their original row ids, and no physical _ROW_ID column is ever written, so row ids stay derived from firstRowId + position, exactly like the files produced by DataEvolutionDeleteRewriter in #8182

@kerwin-zk It's too troublesome to maintain the index after deletion. Maybe the correct solution is deletion-vector. Can you just support update in this PR?

@kerwin-zk kerwin-zk force-pushed the spark-v2-dml-data-evolution branch from c5d12ff to bcbbfb0 Compare June 17, 2026 13:34
@kerwin-zk kerwin-zk changed the title [spark] Support V2 DELETE and UPDATE for data evolution tables [spark] Support V2 UPDATE for data evolution tables Jun 17, 2026
@kerwin-zk kerwin-zk force-pushed the spark-v2-dml-data-evolution branch from bcbbfb0 to 55f0db4 Compare June 17, 2026 14:02
@kerwin-zk kerwin-zk force-pushed the spark-v2-dml-data-evolution branch from 55f0db4 to 1927c17 Compare June 18, 2026 04:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@JingsongLi JingsongLi JingsongLi left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@kerwin-zk @JingsongLi