Runtime and core load discrepancies depending on data handover to future_apply

[hloefflerwirth]

Hey all, I try to implement some parallel processing into my R package and encountered an interesting issue.

Setup:

• R 4.3 was used on a Linux computer, and R 4.2 on another Windows computer

• I use the future functions, however the same behaviour was also shown by the parallel package

• there is a global variable mat, which stores a large data matrix as demonstrator

• there are two almost identical functions, which call future_apply for some parallelized dummy calculations on mat

• funA receives mat as function parameter

• funB accesses mat in terms of global variable (not as function parameter)

• a third test is performed simply calling future_apply outside of a function

library(future)       # version 1.34.0
library(future.apply) # version 1.11.2

plan(multisession, workers=4)


mat <- matrix( 1:100, nrow=40000, ncol=1000)



funA <- function(mat)
{
  future_apply( mat, 1, function(x) sqrt(x[1:500]/x[501:1000]) )
}

t1 <- Sys.time()
dummy <- funA(mat)
Sys.time()-t1
# takes ~3 seconds




funB <- function()
{
  future_apply( mat, 1, function(x) sqrt(x[1:500]/x[501:1000]) )
}


t1 <- Sys.time()
dummy <- funB()
Sys.time()-t1
# takes less than 2 seconds, really uses multiple cores



t1 <- Sys.time()
dummy <- future_apply( mat, 1, function(x) sqrt(x[1:500]/x[501:1000]) )
Sys.time()-t1
# takes less than 2 seconds, really uses multiple cores

Observation:

• funA(mat) takes about 3 seconds, funB() takes less than 2 seconds
• this discrepancy is much more pronounced in 'real data calculation' than in this minimal example
• I also observed that funA(mat) not realy uses multiple cores, while funB() produces nicely distributed load on the cores, especially in 'real data calculation'

I thought that the problem has something to do with data access and scope.
However, also creating local copies of the data matrix within the functions did not change anything.
Still it is a miracle to me why the computation of funA(mat) is not distributed across the cores

This example uses future_apply(), but also parApply() from the parallel package shows a similar behaviour.

I played around with these scripts for a week and tried a lot of variations of the code. E.g. to set environment of future_apply and funA to the global environment, and also the environment of future_apply to the local environment of funA. But still the problem persists.

So, I realy like to understand a bit what is going on there 'under the hood' and eventually make funA perform similar to funB, but still receiving the data in terms of its function parameters.

Thanks for your thoughts and help :)