[QST]!NOT_FOUND <InMemoryTableScanExec> cannot run on GPU #5398
|
What is your question?
Most of my operations were based on Spark SQL. Specially, dataframe.cache() were applied for in-memory acceleration. My question is that does this warning represent that it can not scan GPU in-memory data or datasets haven't been cached to GPU? |
Replies: 3 comments
|
We are still working on optimizing cache and persist for the GPU see #11 Because of how the code works in spark it is going to require changes to spark to be able to even do things like optimizing the compression of the data using the GPU. Currently, this is being worked on as a part of apache/spark#29067. But even after those changes, it would not be enough to cache the data in GPU memory. We might be able to hack something together to make that work, but likely even further changes would be required to spark and the block manager to be able to do something like that. |
We are still working on optimizing cache and persist for the GPU see #11
Because of how the code works in spark it is going to require changes to spark to be able to even do things like optimizing the compression of the data using the GPU. Currently, this is being worked on as a part of apache/spark#29067. But even after those changes, it would not be enough to cache the data in GPU memory. We might be able to hack something together to make that work, but likely even further changes would be required to spark and the block manager to be able to do something like that.