feat: Add full-featured Iceberg MERGE INTO conditional merges support and argument validation by pedrorfdez · Pull Request #3201 · aws/aws-sdk-pandas

pedrorfdez · 2025-09-10T18:40:08Z

Feature or Bugfix

Feature

Detail

Implemented full support for Iceberg MERGE INTO operations in _write_iceberg.py/to_iceberg
Added new parameters for:
- merge_on_clause: Custom ON statement in the MERGE INTO ... USING ... ON [custom_expression] to allow <, <=, > and >= operators. Until now, only column equality was allowed. Risk of having more than one match in target table is warned in stringdocs.
- merge_condition: Added new accepted value conditional_merge
- merge_conditional_clauses: List of dictionaries specifying custom conditional clauses for the MERGE INTO statement.
  Each dictionary should have:
  - 'when': One of ['MATCHED', 'NOT MATCHED', 'NOT MATCHED BY SOURCE'] (case-insensitive)
  - 'action': One of ['UPDATE', 'DELETE', 'INSERT'] (case-insensitive)
  - 'condition': (optional) Additional SQL condition for the clause
  - 'columns': (optional) List of columns to update or insert
  Used only when merge_condition is 'conditional_merge'.
Added argument validation for mutually exclusive and required parameters merge_cols, merge_on_clause, merge_match_nulls, merge_condition, merge_conditional_clauses,
Added and updated unit tests to cover new validation logic and merge scenarios.
Updated docstrings for new parameters and behaviors.
Backward compatibility

Relates

This is the first draft of the implementation, feel free to suggest any changes in the approach. I am open to suggestions.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

pedrorfdez · 2025-09-10T19:12:37Z

Currently failing 1 Athena test:

FAILED tests/unit/test_athena_iceberg.py::test_to_iceberg_conditional_merge_happy_path - AssertionError: Attributes of DataFrame.iloc[:, 0] (column name="id") are different

Attribute "dtype" are different
[left]: int64
[right]: Int64

Plan to fix on next commit.

jaidisido · 2025-10-30T13:45:00Z

AWS CodeBuild CI Report

CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
Commit ID: e7acaf0
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

kukushking · 2025-10-30T13:45:56Z

Hi @pedrorfdez it looks like there is an error in test_to_iceberg_conditional_merge_happy_path:

FAILED tests/unit/test_athena_iceberg.py::test_to_iceberg_conditional_merge_happy_path - AssertionError: Attributes of DataFrame.iloc[:, 0] (column name="id") are different
--
 
Attribute "dtype" are different
[left]:  int64
[right]: Int64
= 1 failed, 2312 passed, 72 skipped, 9 xfailed, 4 xpassed, 143 warnings, 1 rerun in 729.43s (0:12:09) =

jaidisido · 2025-10-30T14:47:17Z

AWS CodeBuild CI Report

CodeBuild project: GitHubDistributedCodeBuild6-jWcl5DLmvupS
Commit ID: e7acaf0
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

awswrangler/athena/_write_iceberg.py

github-actions · 2026-01-02T18:06:31Z

Marking this pull request as stale due to inactivity. This helps our maintainers find and focus on the active pull requests.

pedrorfdez · 2026-02-02T16:25:07Z

I will continue working on this again, thanks for the in-depth review @inbar-beehero will check it out.

kukushking · 2026-02-02T16:36:34Z

AWS CodeBuild CI Report

CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
Commit ID: 04ad6e9
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

kukushking · 2026-02-02T17:05:38Z

AWS CodeBuild CI Report

CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
Commit ID: 7592357
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

…_sql_query dtypes

kukushking · 2026-02-02T17:38:41Z

AWS CodeBuild CI Report

CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
Commit ID: 3cdbe2c
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

kukushking · 2026-02-02T17:40:51Z

AWS CodeBuild CI Report

CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
Commit ID: fece82e
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

…bject) to match athena's read_sql_query dtype conversion

kukushking · 2026-02-02T18:00:20Z

AWS CodeBuild CI Report

CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
Commit ID: 953f887
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

kukushking · 2026-02-03T09:10:25Z

AWS CodeBuild CI Report

CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
Commit ID: f9255aa
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

pedrorfdez · 2026-02-03T09:12:24Z

Although CI CD tests passed, I'm still working on a few refinements in the code, so not ready to review yet

kukushking · 2026-02-03T10:10:17Z

AWS CodeBuild CI Report

CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
Commit ID: 98e9f5b
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

kukushking · 2026-02-03T10:14:55Z

AWS CodeBuild CI Report

CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
Commit ID: 23ccfb7
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

pedrorfdez · 2026-02-03T11:12:55Z

Pull request ready to review

…ses on 'conditional_merge' merge condition

kukushking · 2026-02-03T11:45:34Z

AWS CodeBuild CI Report

CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
Commit ID: 7b00a35
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

github-actions · 2026-04-04T12:12:30Z

Marking this pull request as stale due to inactivity. This helps our maintainers find and focus on the active pull requests.

kukushking · 2026-04-07T18:19:47Z

AWS CodeBuild CI Report

CodeBuild project: GitHubDistributedCodeBuild6-jWcl5DLmvupS
Commit ID: 7f854cd
Result: FAILED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

kukushking · 2026-04-07T18:35:00Z

AWS CodeBuild CI Report

CodeBuild project: GitHubCodeBuild8756EF16-4rfo0GHQ0u9a
Commit ID: 7f854cd
Result: SUCCEEDED
Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

pedrorfdez changed the title ~~Add full-featured Iceberg MERGE INTO conditional merges support and argument validation~~ feat: Add full-featured Iceberg MERGE INTO conditional merges support and argument validation Sep 10, 2025

This comment was marked as outdated.

Sign in to view

pedrorfdez marked this pull request as ready for review September 11, 2025 17:18

This comment was marked as outdated.

Sign in to view

inbar-beehero approved these changes Nov 3, 2025

View reviewed changes

github-actions bot added the closing-soon label Jan 2, 2026

kukushking removed the closing-soon label Jan 7, 2026

pedrorfdez added 8 commits February 2, 2026 17:08

Added necesary parameters to 'to_iceberg' method to expand functionality

21bfee0

Removed unused import

d58094e

Added validation for new parameters complying with new logic

9072ac5

Draft implementation done. Pending tests and verify optimal approach

4aa6eb1

Added additional validation and removed unhandled action.

0a8861f

Added new params to python docstrings

556094e

Fixed backslash in f-string

5d03da1

Added tests

93c9698

pedrorfdez added 2 commits February 2, 2026 17:11

reformat: small reformat

ba9c522

refactor: fix mypy typecheck errors

04ad6e9

pedrorfdez force-pushed the feature/iceberg-merge-into-full-functionality branch from e7acaf0 to 04ad6e9 Compare February 2, 2026 16:21

fix: replace truthiness check with explicit None comparison

7592357

pedrorfdez added 2 commits February 2, 2026 18:23

fix: conditional merge happy path to use Int64 to match athena's read…

3cdbe2c

…_sql_query dtypes

reformat: ruff

fece82e

fix: in happy path test case, convert pandas StringDType to string (o…

953f887

…bject) to match athena's read_sql_query dtype conversion

re-run workflow

f9255aa

pedrorfdez added 2 commits February 3, 2026 10:54

refactor: add helper function '_validate_merge_conditional_clauses'

98e9f5b

reformat: ruff formatting

23ccfb7

pedrorfdez added 2 commits February 3, 2026 12:26

fix: case-insensitive when and action clauses in _merge_iceberg

f6870b3

test: add test case to evaluate case insensitive when and action clau…

7b00a35

…ses on 'conditional_merge' merge condition

github-actions bot added the closing-soon label Apr 4, 2026

Merge branch 'main' into feature/iceberg-merge-into-full-functionality

7f854cd

github-actions bot removed the closing-soon label Apr 7, 2026

Conversation

pedrorfdez commented Sep 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Feature or Bugfix

Detail

Relates

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

pedrorfdez commented Sep 10, 2025

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

jaidisido commented Oct 30, 2025

AWS CodeBuild CI Report

Uh oh!

kukushking commented Oct 30, 2025

Uh oh!

jaidisido commented Oct 30, 2025

AWS CodeBuild CI Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Jan 2, 2026

Uh oh!

pedrorfdez commented Feb 2, 2026

Uh oh!

kukushking commented Feb 2, 2026

AWS CodeBuild CI Report

Uh oh!

kukushking commented Feb 2, 2026

AWS CodeBuild CI Report

Uh oh!

kukushking commented Feb 2, 2026

AWS CodeBuild CI Report

Uh oh!

kukushking commented Feb 2, 2026

AWS CodeBuild CI Report

Uh oh!

kukushking commented Feb 2, 2026

AWS CodeBuild CI Report

Uh oh!

kukushking commented Feb 3, 2026

AWS CodeBuild CI Report

Uh oh!

pedrorfdez commented Feb 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kukushking commented Feb 3, 2026

AWS CodeBuild CI Report

Uh oh!

kukushking commented Feb 3, 2026

AWS CodeBuild CI Report

Uh oh!

pedrorfdez commented Feb 3, 2026

Uh oh!

kukushking commented Feb 3, 2026

AWS CodeBuild CI Report

Uh oh!

github-actions bot commented Apr 4, 2026

Uh oh!

kukushking commented Apr 7, 2026

AWS CodeBuild CI Report

Uh oh!

kukushking commented Apr 7, 2026

AWS CodeBuild CI Report

Uh oh!

Reviewers

Assignees

pedrorfdez commented Sep 10, 2025 •

edited

Loading

pedrorfdez commented Feb 3, 2026 •

edited

Loading