Skip to content

JoinWith operations that can introduce nulls (leftJoin, rightJoin, etc) produce FrameColumns with misleading schema resulting in potential column not found exceptions #2040

Description

@koperagen

Repro:

val l = (0..10).toDataFrame {
    "id" from { it }
}
val r = (0..5).toDataFrame {
    "id" from { it }
    "frame" from {
        (0..10).toDataFrame {
            "col" from { it }
        }
    }
}
val res = l.leftJoinWith(r) { id == right.id }
res.schema().print()
res.frame[6].col

Actual:

id: Int
id1: Int?
frame: *
    col: Int

Exception in thread "main" java.lang.IllegalStateException: Column not found exception in the generated DataFrame extension property 'col': Column not found: 'col'. See https://kotlin.github.io/dataframe/data-schemas-and-extension-properties-troubleshooting.html for more information.
	at org.jetbrains.kotlinx.dataframe.exceptions.ExtensionPropertyExceptionKt.handleExtensionPropertyException(extensionPropertyException.kt:32)
	at JoinFrameKt$main$res$1$DataFramePropertiesScope1.getCol(joinFrame.kt)
	at JoinFrameKt.main(joinFrame.kt:18)
	at JoinFrameKt.main(joinFrame.kt)

Both runtime and compile schemas tell us there's col in frame, but it's not always the case, and simply iterating over res can lead to an unexpected column not found exception.

Image

Root cause:
Potential fix touches broader issue. joinWithImpl uses createByInference, which produces this invalid result.

else -> DataColumn.createByInference(generator.names[index], values)

Smaller repro that would involve only createByInference will be

fun main() {
    val res = dataFrameOf("frame" to columnOf(null, dataFrameOf("col" to columnOf(1))))
    res.cast<DataFrameOf_42>().frame[0].col
}

@DataSchema
data class DataFrameOf_42(
    val frame: List<Frame>
) {
    @DataSchema
    data class Frame(
        val col: Int
    )
}

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions