What happens?
Java table functions can declare regular result columns and enable projection pushdown, but they cannot declare virtual columns.
This is particularly relevant for queries such as:
SELECT count(*) FROM java_table_function()
No user-visible column is required to evaluate this query. However, because the Java table function cannot advertise DuckDB's internal empty virtual column, DuckDB falls back to projecting the first regular result column.
In a minimal test with projection pushdown enabled, init() receives:
columnCount=1
columnIndex=0
The function must therefore produce the first physical column even though the query only needs the number of rows.
Why is this needed?
Some Java table functions read from external cursors, remote services, files, or other expensive sources. Producing a regular column may require decoding, allocating, converting, or transferring values.
For COUNT(*), the function should be able to advance the source and return the chunk cardinality without materializing an otherwise unused payload column.
More generally, exposing virtual columns would allow Java table functions to provide metadata or synthetic columns that are not part of the physical source.
Reproduction
DuckDBFunctions.tableFunction()
.withName("count_star_projection_probe")
.withProjectionPushdown()
.withFunction(new DuckDBTableFunction<Object, AtomicBoolean, Object>() {
@Override
public Object bind(DuckDBTableFunctionBindInfo info) {
info.addResultColumn("expensive_payload", String.class);
return null;
}
@Override
public AtomicBoolean init(DuckDBTableFunctionInitInfo info) {
System.out.println("columnCount=" + info.getColumnCount());
System.out.println("columnIndex=" + info.getColumnIndex(0));
return new AtomicBoolean();
}
@Override
public long apply(
DuckDBTableFunctionCallInfo info,
DuckDBDataChunkWriter output) {
AtomicBoolean done = info.getInitData();
if (done.getAndSet(true)) {
return 0;
}
// Required today even though COUNT(*) does not use this value.
for (long row = 0; row < 3; row++) {
output.vector(0).setString(row, "payload");
}
return 3;
}
})
.register(connection);Running:
SELECT count(*) FROM count_star_projection_probe();
currently reports:
columnCount=1
columnIndex=0
Expected behavior
The Java table-function API should provide a way to declare supported virtual columns, including the internal empty projection used for COUNT(*).
When that capability is available, a query that does not reference any regular column should not force the function to materialize one. The function should be able to produce only the row count for each output chunk.
C API dependency
The DuckDB core already supports TableFunction::get_virtual_columns and uses COLUMN_IDENTIFIER_EMPTY when resolving an empty projection. However, the public C table-function API does not appear to expose a way to register or return virtual columns.
I understand that implementing this in duckdb-java may therefore require a C API addition first.
What happens?
Java table functions can declare regular result columns and enable projection pushdown, but they cannot declare virtual columns.
This is particularly relevant for queries such as:
No user-visible column is required to evaluate this query. However, because the Java table function cannot advertise DuckDB's internal empty virtual column, DuckDB falls back to projecting the first regular result column.
In a minimal test with projection pushdown enabled, init() receives:
columnCount=1
columnIndex=0
The function must therefore produce the first physical column even though the query only needs the number of rows.
Why is this needed?
Some Java table functions read from external cursors, remote services, files, or other expensive sources. Producing a regular column may require decoding, allocating, converting, or transferring values.
For COUNT(*), the function should be able to advance the source and return the chunk cardinality without materializing an otherwise unused payload column.
More generally, exposing virtual columns would allow Java table functions to provide metadata or synthetic columns that are not part of the physical source.
Reproduction
Running:
currently reports:
columnCount=1
columnIndex=0
Expected behavior
The Java table-function API should provide a way to declare supported virtual columns, including the internal empty projection used for COUNT(*).
When that capability is available, a query that does not reference any regular column should not force the function to materialize one. The function should be able to produce only the row count for each output chunk.
C API dependency
The DuckDB core already supports TableFunction::get_virtual_columns and uses COLUMN_IDENTIFIER_EMPTY when resolving an empty projection. However, the public C table-function API does not appear to expose a way to register or return virtual columns.
I understand that implementing this in duckdb-java may therefore require a C API addition first.