toJson() drops row objects when a real "value" column contains only nulls

[chimission]

Description

DataFrame.toJson() incorrectly serializes a multi-column DataFrame as an array of null values when a column named value contains only nulls, even though other columns contain valid data.

Minimal reproduction

import org.jetbrains.kotlinx.dataframe.api.dataFrameOf
import org.jetbrains.kotlinx.dataframe.io.toJson

fun main() {
    val df =
        dataFrameOf(
            "value" to listOf<Double?>(null, null),
            "name" to listOf("Alice", "Bob"),
        )

    println(df.toJson())
}

Actual result

[null,null]

Expected result

[
  {
    "value": null,
    "name": "Alice"
  },
  {
    "value": null,
    "name": "Bob"
  }
]

Environment

• Kotlin: 2.2.21
• Kotlin DataFrame: 1.0.0-Beta5
• JDK: 21
• Gradle: 9.4.1
• OS: Windows 11

[Jolanrensen]

Hi, thanks for letting us know!

I have a hunch that this is intended behavior. It's namely the reverse of JSON.TypeClashTactic.ARRAY_AND_VALUE_COLUMNS https://kotlin.github.io/dataframe/read.html#manage-type-clashes.

I need some time to think about the original purpose of this function and why it was written this way.

From what I can see, the original logic is: A column is considered the value column in JSON if it's named value*, it's a ValueColumn and for each row it holds that it is null and/or all the other cells in the same row are null. (In other words, if in the value column, the value is not null, it must be the only non-null value in the row)

Your PR changed it like: A column is ... and for each row it holds that all the other cells in the same row are null regardless of what value holds.

Am I correct?

[chimission]

@Jolanrensen
Yes, your understanding of the previous version is correct. After reviewing the original purpose, I agree that my first change was too broad.
I have now narrowed the fix to the all-null edge case:

• If the candidate value* column contains at least one non-null value, the original validation logic is preserved unchanged. This keeps the existing ARRAY_AND_VALUE_COLUMNS mixed-type behavior.
• If the entire candidate column is null, all other cells must also be null for it to be recognized as the special value column. Otherwise, it is treated as a regular business column and the rows are encoded as objects.
  In other words, the additional restriction only prevents an all-null candidate from passing the original condition vacuously.
  I applied the same adjustment to extractArrayColumn(), as it had the same all-null edge case.
  There is still an unavoidable ambiguity when every cell in a multi-column frame is null: without preserving provenance from readJson(), the writer cannot distinguish a generated special column from a user-created one. In that case, I kept the existing scalar behavior ([null, null]) for compatibility.

I also added regression tests covering:

• all-null and partially-null business value columns;
• generated scalar and all-null scalar value columns;
• mixed JSON retaining its generated value column;
• value1, array, and array1 cases;
• all-null special columns with no data in the other columns.

[Jolanrensen]

If the entire candidate column is null, all other cells must also be null for it to be recognized as the special value column.

@chimission After thinking about it for a while and rewriting the logic a bit to see it from other angles, I'm still not 100% convinced about this rule. It's mostly because I don't think there's a way that DataFrame.readJsonStr() can generate an all-nulls 'value' column:

[{"placeholder": 123}, 1, null]

becomes

     placeholder  value
            Int?   Int?
            ----   ----
  0          123   null
  1         null      1
  2         null   null

but

[{"placeholder": 123}, null, null]

becomes simply

     placeholder
            Int?
            ----
  0          123
  1         null
  2         null

(null is considered part of the object)
So with that logic, an all-null column named "value" can never have been a 'value' column.

I wonder which case we would solve by adding that extra rule

[chimission]

@Jolanrensen

The goal was to keep the fix focused on the false positive we actually observed: an all-null value column being treated as special even though other columns still contain data. I wanted to avoid broadening the scope of this PR and potentially introducing incompatible behavior in other, more ambiguous cases.

The post-transformation case I had in mind was:

val df = DataFrame.readJsonStr(
    """[{"placeholder": 123}, 1, null]"""
)

This initially produces:

placeholder  value
        123   null
       null      1
       null   null

Here, value is the special column created for the scalar value 1.

If only the last row is subsequently selected:

val nullRow = df[2..2]

the operation preserves both columns, producing:

placeholder  value
       null   null

The value column is now entirely null, even though it originated as a special column.

With the previous behavior, nullRow.toJson() returns:

[null]

If every all-null value candidate is treated as a regular column, it would instead return:

[{"placeholder":null,"value":null}]

This seems to point to a broader limitation: once the parsing-only UnnamedColumn marker is gone, toJson() has no reliable way to tell a generated value column from a user-defined one based only on its name, type, and contents. Preserving that information through DataFrame transformations would be a larger change, so for this PR I’d prefer to keep the scope focused on the concrete data-loss issue.

[Jolanrensen]

@chimission Thanks for the clarification.

However, the data loss is due to #2045. So, in your example, the all-null column value is correctly recognized as 'value' column, because it disappears. If it were not recognized as 'value' column and be treated as a 'regular' column, we'd indeed get [{"placeholder":null,"value":null}].

So with that in mind, to let that all-null 'value' column not disappear, an all-null column named "value" should NOT be recognized as 'value' column, correct?

For some more clarification, when converting a JSON array to DataFrame, we grab all objects inside and create columns from the keys, filling in the values on the rows. We consider these 'regular' columns.
If that array contains loose values or arrays, we create a 'value' column or 'array' column containing these loose values/arrays. The values in the 'normal' column will be null in the places the 'value'- or 'array' column has a value.

[chimission]

@Jolanrensen
Thanks, I understand your point now, and I agree that an all-null value* column should be treated as a regular column.