Skip to content

fix: any and all functions should not have default frame or order_by#11930

Draft
ssabdb wants to merge 5 commits intoibis-project:mainfrom
ssabdb:any-all-fix
Draft

fix: any and all functions should not have default frame or order_by#11930
ssabdb wants to merge 5 commits intoibis-project:mainfrom
ssabdb:any-all-fix

Conversation

@ssabdb
Copy link
Copy Markdown
Contributor

@ssabdb ssabdb commented Feb 17, 2026

Description of changes

The any and all operators should not have a default ordering or set a restricted 'current row' frame by default, because they are not operators which by default should be ordered.

I noticed a bug in bigquery where filtering e.g.
table.filter((.a == "hello").any().over(group_by=(.x)))

produces an incorrect window clause, including
(PARTITION BY t0.x order by null asc ROWS between unbounded preceding and current row)

which leads to inconsistent and incorrect results.

The expected behaviour is to produce no frame or order by, e.g.

(PARTITION BY t0.x ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING)

This is because ordering by null and the current row results in a undefined order. I suspect this will probably apply to multiple backends.

The fix involves modifying exclude_unsupported_window_frame_from_ops in rewrites.py to exclude Any and All.

Closes #11931

@github-actions github-actions bot added tests Issues or PRs related to tests bigquery The BigQuery backend sql Backends that generate SQL labels Feb 17, 2026
@ssabdb
Copy link
Copy Markdown
Contributor Author

ssabdb commented Feb 18, 2026

Since the change is to the base sql compiler rewrites.py, should the test be moved to ibis/backends/sql/tests/test_compiler.py?

@NickCrews
Copy link
Copy Markdown
Contributor

Since the change is to the base sql compiler rewrites.py, should the test be moved to ibis/backends/sql/tests/test_compiler.py?

Almost, I think that file is just for testing the base Sql compiler, when ideally we would actually want to test this for each backend. So, it should go in ibis/backends/tests/test_sql.py, eg toss this at the bottom

@pytest.mark.parametrize("backend_name", _get_backends_to_test())
@pytest.mark.notimpl(["polars"], raises=ValueError, reason="not a SQL backend")
def test_window_filter_any(backend_name, snapshot):
    t = ibis.table([("a", "string"), ("x", "int64")], name="t")
    expr = t.filter((_.a == "hello").any().over(group_by=_.x))
    sql = ibis.to_sql(expr, dialect=backend_name)
    snapshot.assert_match(sql, "out.sql")

Second, can you actually explain what the exclude_unsupported_window_frame_from_ops rewrite rule DOES? I am decently confident in this change because the tests pass, but I don't have a good grip on what the other side-effects of this change would be. If you understand, it would be great if you could add a comment to the rewrite func

Copy link
Copy Markdown
Contributor

@NickCrews NickCrews left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See my above comment, move the test to check all backends, and ideally explain the rewrite rule

@ssabdb ssabdb marked this pull request as draft February 26, 2026 23:41
@ssabdb
Copy link
Copy Markdown
Contributor Author

ssabdb commented Feb 27, 2026

Many thanks for the comments.

I'm also confused by what some of these rewrites functions are trying to do with window functions and on moving the tests noticed that some backends, e.g. mysql were still creating a window function with an order by null.

I think exclude_unsupported_window_frame_from_ops is designed to exclude window functions from navigation functions, e.g. lag, lead where there is no meaning to a frame. Any and All operators were erroneously included.

The function exclude_unsupported_window_frame_from_rank also silently injects a null order by if it's missing for a rank and the rewrite_empty_order_by_window always injects a null order for all window functions for certain backends if they are missing, e.g. mysql if it makes sense or not for the operator being used.

My conclusion is that these functions need a small amount of rationalization, probably to separate one to zero index transformation - there seem to be several of these - from forcing default order_by which feels like it should be consistent across all backends?

I've pushed some (not ready for merging) changes which tries to work on the rewrites only. I suspect it will break a few things.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bigquery The BigQuery backend sql Backends that generate SQL tests Issues or PRs related to tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: any and all operators incorrectly apply default frame and sort order with window function

2 participants