[SPARK-59425][PYTHON] Introduce EvalTypeHandler pipeline and migrate scalar Arrow UDF - #58729

Open

Yicong-Huang wants to merge 19 commits into

apache:masterfrom

Yicong-Huang:eval-handler-base

Commits on Sep 11, 2026

refactor: introduce EvalTypeHandler pipeline and migrate scalar Arrow UDF

Yicong-Huang
and
Isaac
committed
cc002ee
refactor: move eval type handlers into pyspark.sql.pandas.eval_type_handlers

Yicong-Huang
and
Isaac
committed
d518f66
refactor: split eval handlers into pyspark.sql.eval_handlers package (arrow/pandas)

Yicong-Huang
and
Isaac
committed
0cf5afd
refactor: make arrow handler module private and drop empty pandas module

Yicong-Huang
and
Isaac
committed
029e45c
refactor: extract verify helpers to eval_handlers.verification and make registry public

Yicong-Huang
and
Isaac
committed
0ed0d0f
refactor: expose handler serializer as a property instead of select_serializer()

Yicong-Huang
and
Isaac
committed
41332db
refactor: reduce handler contract to run() plus serializer

Yicong-Huang
and
Isaac
committed
219d4dd
refactor: tighten handler type hints (RunnerConf/EvalConf/Serializer, no ignores)

Yicong-Huang
and
Isaac
committed
1316bdf
refactor: use __all__ re-export and unquote runtime-available type hints

Yicong-Huang
and
Isaac
committed
f7098e7
refactor: move RunnerConf and EvalConf to worker_util next to their Conf base

Yicong-Huang
and
Isaac
committed
5b33e4e
refactor: define Arrow stream type aliases in eval_handlers, drop pandas._typing dep

Yicong-Huang
and
Isaac
committed
0d34021
refactor: concise comments on new handler code and split type aliases into _typing

Yicong-Huang
and
Isaac
committed
d148c5a
refactor: drop __all__ from internal handler package, register via side-effect import

Yicong-Huang
and
Isaac
committed
0c919d5
refactor: re-export concrete handlers from package so callers avoid private submodules

Yicong-Huang
and
Isaac
committed
b2cdcb5
Merge upstream/master; adapt handler dispatch to the (func, serializer) runner contract

Yicong-Huang
and
Isaac
committed
6da17e5

Commits on Sep 12, 2026

fix: use assert instead of raise AssertionError to satisfy custom-errors lint

Yicong-Huang
and
Isaac
committed
f547e5a

Commits on Sep 14, 2026

Merge upstream/master

Yicong-Huang
and
Isaac
committed
5de2547

Commits on Sep 15, 2026

refactor: extract eval handler base classes into _base to break import cycle

Yicong-Huang
and
Isaac
committed
2123fe3
Merge upstream/master; port new RunnerConf arrow output-batch-max-bytes property into worker_util

Yicong-Huang
and
Isaac
committed
7b66329