Commit 7a5d7f1
authored
[data] revert(test_gate): remove test gate from test_write_delta.py (#65588)
## Description
In the tests "test_write_delta.py", we were doing a round trip assertion
on write_delta -> read_delta. These assertions were originally failing
in CI with pyarrow version 17 so we added assertions to skip certain
tests if pyarrow's available version was < "19.0.0".
When attempting to fix the underlying issue, it is not getting
reproduced locally using pyarrow v17 or above (supported versions) on
both x86_64 and arm64 machines using python 3.10/3.12; Hence I'm raising
this pull request to revert the test gates.
Script used for local repro -
```python3
import os
import sys
import tempfile
import deltalake # noqa: E402
import pyarrow # noqa: E402
import ray # noqa: E402
# One typed partition value and one null, which Delta stores under the
# sentinel directory name "year=__HIVE_DEFAULT_PARTITION__".
ROWS = [{"id": 1, "year": 2024}, {"id": 2, "year": None}]
def main() -> int:
ray.data.DataContext.get_current().enable_progress_bars = False
print(
f"ray {ray.__version__} | "
f"pyarrow {pyarrow.__version__} | "
f"deltalake {deltalake.__version__}"
)
with tempfile.TemporaryDirectory() as tmp_dir:
table_uri = os.path.join(tmp_dir, "delta_table")
ray.data.from_items(ROWS).write_delta(table_uri, partition_by=["year"])
print("on disk: ", sorted(os.listdir(table_uri)))
# take_all() gives no ordering guarantee across blocks; sort to compare.
got = sorted(ray.data.read_delta(table_uri).take_all(), key=lambda r: r["id"])
print("read back:", got)
print("expected: ", ROWS)
if got != ROWS:
print(
"FAIL: partition values did not survive the round trip",
file=sys.stderr,
)
return 1
print("OK")
return 0
if __name__ == "__main__":
sys.exit(main())
```
which generates the following output -
```log
ray 3.0.0.dev0 | pyarrow 17.0.0 | deltalake 1.6.2
2026-08-20 15:34:17,187 INFO worker.py:2015 -- Started a local Ray instance. View the dashboard at http://127.0.0.1:8265
2026-08-20 15:34:18,148 WARNING __init__.py:28 -- Progress bars disabled. To enable, set `ray.data.DataContext.get_current().enable_progress_bars = True`.
on disk: ['_delta_log', 'year=2024', 'year=__HIVE_DEFAULT_PARTITION__']
read back: [{'id': 1, 'year': 2024}, {'id': 2, 'year': None}]
expected: [{'id': 1, 'year': 2024}, {'id': 2, 'year': None}]
OK
```
## Related issues
Relates to #65143
Signed-off-by: Dhruvil Shah <dhruvil.shah@anyscale.com>1 parent 6e0abdd commit 7a5d7f1
1 file changed
Lines changed: 0 additions & 37 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
41 | 41 | | |
42 | 42 | | |
43 | 43 | | |
44 | | - | |
45 | | - | |
46 | | - | |
47 | | - | |
48 | | - | |
49 | | - | |
50 | | - | |
51 | | - | |
52 | | - | |
53 | | - | |
54 | | - | |
55 | | - | |
56 | | - | |
57 | | - | |
58 | | - | |
59 | | - | |
60 | | - | |
61 | | - | |
62 | | - | |
63 | | - | |
64 | | - | |
65 | | - | |
66 | | - | |
67 | | - | |
68 | | - | |
69 | 44 | | |
70 | 45 | | |
71 | 46 | | |
| |||
197 | 172 | | |
198 | 173 | | |
199 | 174 | | |
200 | | - | |
201 | | - | |
202 | 175 | | |
203 | 176 | | |
204 | 177 | | |
205 | 178 | | |
206 | | - | |
207 | 179 | | |
208 | 180 | | |
209 | 181 | | |
| |||
226 | 198 | | |
227 | 199 | | |
228 | 200 | | |
229 | | - | |
230 | | - | |
231 | 201 | | |
232 | 202 | | |
233 | 203 | | |
| |||
270 | 240 | | |
271 | 241 | | |
272 | 242 | | |
273 | | - | |
274 | | - | |
275 | 243 | | |
276 | 244 | | |
277 | 245 | | |
| |||
330 | 298 | | |
331 | 299 | | |
332 | 300 | | |
333 | | - | |
334 | | - | |
335 | 301 | | |
336 | 302 | | |
337 | 303 | | |
338 | | - | |
339 | 304 | | |
340 | 305 | | |
341 | 306 | | |
| |||
368 | 333 | | |
369 | 334 | | |
370 | 335 | | |
371 | | - | |
372 | | - | |
373 | 336 | | |
374 | 337 | | |
375 | 338 | | |
| |||
0 commit comments