Skip to content

Commit b49ed02

Browse files
Frank Bertschpan3793
authored andcommitted
[KYUUBI #7106] Make response.results.columns optional
### Why are the changes needed? Bugfix. Spark 3.5 is returning `None` for `response.results.columns`, while Spark 3.3 returned actual values. The response here: https://github.com/apache/kyuubi/blob/master/python/pyhive/hive.py#L507 For a query that does nothing (mine was an `add jar s3://a/b/c.jar`), here are the responses I received. Spark 3.3: ``` TFetchResultsResp(status=TStatus(statusCode=0, infoMessages=None, sqlState=None, errorCode=None, errorMessage=None), hasMoreRows=False, results=TRowSet(startRowOffset=0, rows=[], columns=[TColumn(boolVal=None, byteVal=None, i16Val=None, i32Val=None, i64Val=None, doubleVal=None, stringVal=TStringColumn(values=[], nulls=b'\x00'), binaryVal=None)], binaryColumns=None, columnCount=None)) ``` Spark 3.5: ``` TFetchResultsResp(status=TStatus(statusCode=0, infoMessages=None, sqlState=None, errorCode=None, errorMessage=None), hasMoreRows=False, results=TRowSet(startRowOffset=0, rows=[], columns=None, binaryColumns=None, columnCount=None)) ``` ### How was this patch tested? I tested by applying it locally and running my query against Spark 3.5. I was not able to get any unit tests running, sorry! ### Was this patch authored or co-authored using generative AI tooling? No. Closes #7107 from fbertsch/spark_3_5_fix. Closes #7106 13d1440 [Frank Bertsch] Make response.results.columns optional Authored-by: Frank Bertsch <fbertsch@netflix.com> Signed-off-by: Cheng Pan <chengpan@apache.org>
1 parent e769f42 commit b49ed02

1 file changed

Lines changed: 8 additions & 5 deletions

File tree

python/pyhive/hive.py

Lines changed: 8 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -508,14 +508,17 @@ def _fetch_more(self):
508508
_check_status(response)
509509
schema = self.description
510510
assert not response.results.rows, 'expected data in columnar format'
511-
columns = [_unwrap_column(col, col_schema[1]) for col, col_schema in
512-
zip(response.results.columns, schema)]
513-
new_data = list(zip(*columns))
514-
self._data += new_data
511+
has_new_data = False
512+
if response.results.columns:
513+
columns = [_unwrap_column(col, col_schema[1]) for col, col_schema in
514+
zip(response.results.columns, schema)]
515+
new_data = list(zip(*columns))
516+
self._data += new_data
517+
has_new_data = (True if new_data else False)
515518
# response.hasMoreRows seems to always be False, so we instead check the number of rows
516519
# https://github.com/apache/hive/blob/release-1.2.1/service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java#L678
517520
# if not response.hasMoreRows:
518-
if not new_data:
521+
if not has_new_data:
519522
self._state = self._STATE_FINISHED
520523

521524
def poll(self, get_progress_update=True):

0 commit comments

Comments
 (0)