Skip to content

[BugFix] Fix Iceberg equality-delete under-application when a string date partition column is compared with a temporal value (backport #76280)#76293

Merged
wanpengfei-git merged 2 commits into
branch-4.1from
mergify/bp/branch-4.1/pr-76280
Jul 13, 2026
Merged

[BugFix] Fix Iceberg equality-delete under-application when a string date partition column is compared with a temporal value (backport #76280)#76293
wanpengfei-git merged 2 commits into
branch-4.1from
mergify/bp/branch-4.1/pr-76280

Conversation

@mergify

@mergify mergify Bot commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Why I'm doing

Fourth (and last) in the series fixing one root cause across its consumers. When a STRING date
partition column is compared with a temporal value, e.g.

  WHERE c2 = cast('2020-06-15' as date) - interval 1 day

binary-predicate coercion produces CAST(c2 AS DATETIME) = '2020-06-14 00:00:00'. Iceberg's native
pushdown unwraps the cast and compares in the STRING domain, which never equals a 'yyyy-MM-dd'
partition value.

getDeleteFiles pushed the whole predicate through the string-domain converter to filter the
equality-delete-file scan. A CAST(string_partition_col AS DATETIME) = <ts> conjunct therefore
pruned every equality-delete file whose 'yyyy-MM-dd' partition value did not equal the rendered
'...00:00:00' string. IcebergEqualityDeleteRewriteRule then derived no delete schemas, the
equality-delete anti-join was not applied, and already-deleted rows leaked back into the result.
Unlike the other paths, this is a silent correctness bug on reads (stale rows returned).

What I'm doing

Reuse PartitionCastPredicatePruner:

  • Do not push cast-on-string-partition conjuncts to the delete-file scan filter; only the
    non-cast (pushable) conjuncts are converted and pushed.
  • After planning, filter the returned delete files StarRocks-side by their partition values with
    partitionMayMatch (drops a delete file only when its partition value provably cannot satisfy the
    residual). Keeping a possibly-matching delete file is safe (a superset only over-applies equality
    deletes, which match by key and never remove non-matching rows); dropping a needed one is not, so
    the direction matches [BugFix] Fix Iceberg partition pruning when a string date partition column is compared with a temporal value #76068's scan-side pruning.
  • Add deleteFilePartitionValues(DeleteFile, table), which decodes a delete file's partition values
    using the file's own partition spec (a delete file may predate the current spec) and returns an
    empty map on any decoding uncertainty (partition-transform evolution, dropped source column, etc.),
    so partitionMayMatch conservatively keeps the file.

Reproduce

Equality-delete files can only be written by Flink (StarRocks/Spark write position deletes), so the
fixture is created with a Flink upsert:

  -- Flink
  CREATE TABLE ice.db.t_eqdel (
      id INT, c2 STRING, v INT, PRIMARY KEY (id, c2) NOT ENFORCED
  ) PARTITIONED BY (c2) WITH ('format-version'='2', 'write.upsert.enabled'='true');
  INSERT INTO ice.db.t_eqdel VALUES (1, '2020-06-14', 100);
  INSERT INTO ice.db.t_eqdel VALUES (1, '2020-06-14', 200);  -- upsert -> equality delete of (1,100)
  INSERT INTO ice.db.t_eqdel VALUES (2, '2020-06-22', 300);

  -- StarRocks
  SELECT id, c2, v FROM t_eqdel WHERE c2 = cast('2020-06-15' as date) - interval 1 day ORDER BY id, v;
  -- before: 1|2020-06-14|100 AND 1|2020-06-14|200   (deleted v=100 leaks)
  -- after : 1|2020-06-14|200
  -- baseline (no predicate) and the bare-string 'c2 = 2020-06-14' already returned only v=200.

What type of PR is this:

  • BugFix
  • Feature
  • Enhancement
  • Refactor
  • UT
  • Doc
  • Tool

Does this PR entail a change in behavior?

  • Yes, this PR will result in a change in behavior.
  • No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

  • Interface/UI changes: syntax, type conversion, expression evaluation, display information
  • Parameter changes: default values, similar parameters but with different default values
  • Policy changes: use new policy to replace old one, functionality automatically enabled
  • Feature removed
  • Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

  • I have added test cases for my bug fix or my new feature
  • This pr needs user documentation (for new or modified features or behaviors)
    • I have added documentation for my new feature or new function
    • This pr needs auto generate documentation
  • This is a backport pr

Bugfix cherry-pick branch check:

  • I have checked the version labels which the pr will be auto-backported to the target branch
    • 4.1
    • 4.0
    • 3.5

This is an automatic backport of pull request #76280 done by [Mergify](https://mergify.com).

…date partition column is compared with a temporal value (#76280)

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
(cherry picked from commit f62849f)
Stub nativeTbl.spec() so the mocked Iceberg table behaves like a real
one; getDeleteFiles now dereferences spec() via
identityStringPartitionColumns.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: trueeyu <lxhhust350@qq.com>
@wanpengfei-git wanpengfei-git merged commit 0b567c7 into branch-4.1 Jul 13, 2026
28 of 29 checks passed
@wanpengfei-git wanpengfei-git deleted the mergify/bp/branch-4.1/pr-76280 branch July 13, 2026 13:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants