[SPARK-58601][PYTHON][FOLLOWUP] Default mapInBatch acceptAnyIterable to true

### What changes were proposed in this pull request?

Follow-up to SPARK-58601, which tightened the `mapInPandas`/`mapInArrow` return-value contract to require a strict `Iterator` by default and added the internal escape-hatch config `spark.sql.execution.pythonUDF.mapInBatch.legacy.acceptAnyIterable.enabled` (default `false`). This PR flips that config's default to `true`, so the pre-4.3.0 lenient behavior (accepting any iterable such as a returned `list`) remains the default and can be disabled by setting the flag to `false` to opt into strict enforcement. The runtime code in `worker.py` and the config wiring are unchanged apart from the default. The 4.2-to-4.3 PySpark migration-guide note describing the breaking change is removed since there is no longer a default behavior change, and the negative tests that assert a returned `list` is rejected now set the flag to `false` explicitly.

### Why are the changes needed?

Requiring a strict `Iterator` by default is a user-facing breaking change: a `mapInPandas`/`mapInArrow` UDF that returns a non-`Iterator` iterable (e.g. a `list`) previously worked and would start raising `UDF_RETURN_TYPE`. Defaulting the escape hatch to `true` preserves the established behavior for existing workloads while keeping the flag available for users who want the stricter contract, deferring any default change to a later decision.

### Does this PR introduce _any_ user-facing change?

Yes, relative to the unreleased SPARK-58601 change on the unreleased branch. With this PR a `mapInPandas`/`mapInArrow` UDF that returns any iterable (e.g. a `list`) is accepted again by default, as it was before SPARK-58601. Setting `spark.sql.execution.pythonUDF.mapInBatch.legacy.acceptAnyIterable.enabled=false` restores the strict `Iterator`-only contract. Compared to released Spark versions there is no behavior change.

### How was this patch tested?

Updated `test_pandas_map.py` and `test_arrow_map.py`: the negative cases asserting a returned `list` is rejected now run with the flag set to `false`; the existing `*_legacy_accept_any_iterable` tests continue to cover the lenient path.

### Was this patch authored or co-authored using generative AI tooling?

No

Closes #58436 from Yicong-Huang/SPARK-58601-default-true.

Authored-by: Yicong Huang <17627829+Yicong-Huang@users.noreply.github.com>
Signed-off-by: Yicong-Huang <17627829+Yicong-Huang@users.noreply.github.com>

Author: Yicong-Huang

python/docs/source/migration_guide/pyspark_upgrade.rst

@@ -19,10 +19,6 @@
 Upgrading PySpark
 ==================
 
-Upgrading from PySpark 4.2 to 4.3
----------------------------------
-* In Spark 4.3, a ``mapInPandas`` UDF must return an iterator of ``pandas.DataFrame``\s; returning any other iterable such as a ``list`` now raises ``UDF_RETURN_TYPE``, matching the existing ``mapInArrow`` behavior and the declared ``Iterator[...]`` signature. To restore the previous behavior of accepting any iterable for both ``mapInPandas`` and ``mapInArrow``, set ``spark.sql.execution.pythonUDF.mapInBatch.legacy.acceptAnyIterable.enabled`` to ``true``.
-
 Upgrading from PySpark 4.1 to 4.2
 ---------------------------------
 * In Spark 4.2, the minimum supported version for PyArrow has been raised from 15.0.0 to 18.0.0 in PySpark.

python/pyspark/sql/tests/arrow/test_arrow_map.py

@@ -167,9 +167,14 @@ def list_not_iter(_):
         ):
             (self.spark.range(10, numPartitions=3).mapInArrow(bad_iter_elem, "a int").count())
 
-        with self.assertRaisesRegex(
-            PythonException,
-            r"iterator of pyarrow\.RecordBatch.*\blist\b",
+        with (
+            self.sql_conf(
+                {"spark.sql.execution.pythonUDF.mapInBatch.legacy.acceptAnyIterable.enabled": False}
+            ),
+            self.assertRaisesRegex(
+                PythonException,
+                r"iterator of pyarrow\.RecordBatch.*\blist\b",
+            ),
         ):
             (self.spark.range(10, numPartitions=3).mapInArrow(list_not_iter, "a int").count())
 

python/pyspark/sql/tests/pandas/test_pandas_map.py

@@ -232,10 +232,15 @@ def list_not_iter(iterator):
         ):
             (self.spark.range(10, numPartitions=3).mapInPandas(bad_iter_elem, "a int").count())
 
-        with self.assertRaisesRegex(
-            PythonException,
-            "Return type of the user-defined function should be iterator of pandas.DataFrame, "
-            "but is list",
+        with (
+            self.sql_conf(
+                {"spark.sql.execution.pythonUDF.mapInBatch.legacy.acceptAnyIterable.enabled": False}
+            ),
+            self.assertRaisesRegex(
+                PythonException,
+                "Return type of the user-defined function should be iterator of pandas.DataFrame, "
+                "but is list",
+            ),
         ):
             (self.spark.range(10, numPartitions=3).mapInPandas(list_not_iter, "a int").count())
 

python/pyspark/worker.py

@@ -152,7 +152,7 @@ def map_in_batch_legacy_accept_any_iterable(self) -> bool:
         return (
             self.get(
                 "spark.sql.execution.pythonUDF.mapInBatch.legacy.acceptAnyIterable.enabled",
-                "false",
+                "true",
             )
             == "true"
         )

sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala

@@ -5565,7 +5565,7 @@ object SQLConf {
       .version("4.3.0")
       .withBindingPolicy(ConfigBindingPolicy.SESSION)
       .booleanConf
-      .createWithDefault(false)
+      .createWithDefault(true)
 
   val PYTHON_PLANNER_EXEC_MEMORY =
     buildConf("spark.sql.planner.pythonExecution.memory")