Problem
DefaultExpressionEvaluator does not validate that its declared input_schema matches the schema of the actual input batch.
For example, a write context defines:
logical_data_schema = { a, b }physical_data_schema = { p_a, p_b }- Logical-to-physical transform:
first col -> p_a, second col -> p_b
The connector passes a batch ordered as { b=2, a=1 } to evaluate, while the evaluator expects { a, b }. The evaluator can silently produce { p_a=2, p_b=1 }, swapping the values and potentially corrupting written data.
Proposed fix
Long term
DefaultExpressionEvaluator.evaluate should validate input schema vs real data schema with a general rule. Several approaches need exploration, e.g.
- lenient: validate field type + name, when an ancestor is nullable, the sub fields can be missing.
- strict: validate field type + name + nullability + metadata
The final solution maybe sth between strict and lenient
Short term
Validating schemas recursively is non-trivial because some Kernel paths use partial nested schemas, especially for parsed AddFile statistics.
As a first step, validate the actual batch schema against input_schema at the top level, including order, names, and compatible types. Return an error when they do not match.
Note: it should allow extra fields, bc some Kernel code does not provide the full input schema to the evaluator. For example, scan_metadata_from may evaluate scan rows containing optional stats_parsed and partitionValues_parsed columns using only the base scan-row schema.
Problem
DefaultExpressionEvaluatordoes not validate that its declaredinput_schemamatches the schema of the actual input batch.For example, a write context defines:
logical_data_schema = { a, b }physical_data_schema = { p_a, p_b }first col -> p_a,second col -> p_bThe connector passes a batch ordered as
{ b=2, a=1 }toevaluate, while the evaluator expects{ a, b }. The evaluator can silently produce{ p_a=2, p_b=1 }, swapping the values and potentially corrupting written data.Proposed fix
Long term
DefaultExpressionEvaluator.evaluateshould validate input schema vs real data schema with a general rule. Several approaches need exploration, e.g.The final solution maybe sth between strict and lenient
Short term
Validating schemas recursively is non-trivial because some Kernel paths use partial nested schemas, especially for parsed AddFile statistics.
As a first step, validate the actual batch schema against
input_schemaat the top level, including order, names, and compatible types. Return an error when they do not match.Note: it should allow extra fields, bc some Kernel code does not provide the full input schema to the evaluator. For example,
scan_metadata_frommay evaluate scan rows containing optionalstats_parsedandpartitionValues_parsedcolumns using only the base scan-row schema.