library(here) # manage file paths
library(socviz) # data and some useful functions
library(tidyverse) # your friend and mine
library(mirai) # parallel workers for purrrExample 10: Parallel iteration with in_parallel()
Load our packages
A slow task
A bootstrap draws many random samples from the data, with replacement. It then calculates the same statistic on each sample. Each sample is independent of the others, so the task is easy to divide among several processes.
This function draws one bootstrap sample from organdata. It fits a model of the donor rate on consent law and returns the coefficient for presumed consent. The Sys.sleep() call adds a quarter of a second, to make the function slow.
boot_coef <- function(i, data) {
boot_sample <- dplyr::slice_sample(data, prop = 1, replace = TRUE)
fit <- stats::lm(donors ~ consent_law, data = boot_sample)
Sys.sleep(0.25)
tibble::tibble(rep = i, estimate = stats::coef(fit)[["consent_lawPresumed"]])
}
boot_coef(1, organdata)# A tibble: 1 × 2
rep estimate
<dbl> <dbl>
1 1 3.84
The function refers to package functions with ::, for example dplyr::slice_sample(). The parallel workers do not automatically load our packages. The :: prefix tells each worker where to find the function.
Run it sequentially
First, we run 24 replicates with an ordinary map(). Each replicate takes at least 0.25 seconds, so this takes at least six seconds.
system.time(
seq_out <- map(1:24, \(i) boot_coef(i, organdata)) |>
list_rbind()
) user system elapsed
0.100 0.002 6.178
Start the workers
daemons() from the mirai package starts a set of background R processes. These processes are the workers. Do not start more workers than the number of cores in your computer. Here we use four workers, or fewer if the computer has fewer cores.
The seed argument makes the random numbers in the workers reproducible.
n_workers <- min(4, parallel::detectCores() - 1)
daemons(n_workers, seed = 1234)
status()$connections[1] 4
Which process does the work?
Sys.getpid() returns the ID number of the R process that runs it. In an ordinary map(), each call runs in our own R session. So each call returns the same number.
map_int(1:8, \(x) Sys.getpid()) |>
unique()[1] 6897
When we put the function inside in_parallel(), purrr sends each call to a worker. Each call can return a different number.
map_int(1:8, in_parallel(\(x) Sys.getpid())) |>
unique()[1] 6933 6948 6935 6940
Run it in parallel
Next, we run the bootstrap with in_parallel(). The workers do not see the objects in our session. So we must give them each object that the function needs. Here, the function needs boot_coef() and organdata. We supply them as named arguments to in_parallel().
system.time(
par_out <- map(
1:24,
in_parallel(
\(i) boot_coef(i, organdata),
boot_coef = boot_coef,
organdata = organdata
)
) |>
list_rbind()
) user system elapsed
0.004 0.007 1.667
The time is approximately the sequential time divided by the number of workers. There’s some overhead. Each worker must also start and receive the data, so the gain is a little smaller than that.
A common error
If we do not supply an object, the workers cannot find it. The map stops with an error.
map(1:4, in_parallel(\(i) boot_coef(i, organdata)))Error in `map()`:
ℹ In index: 1.
Caused by error in `boot_coef()`:
! could not find function "boot_coef"
Look at the results
The parallel version gives one row per replicate, the same as the sequential version. The random samples are different, so the estimates are different. They have approximately the same distribution.
bind_rows(
Sequential = seq_out,
Parallel = par_out,
.id = "method"
) |>
summarize(
mean = mean(estimate),
sd = sd(estimate),
.by = method
)# A tibble: 2 × 3
method mean sd
<chr> <dbl> <dbl>
1 Sequential 3.24 0.646
2 Parallel 3.32 0.730
Stop the workers
When the parallel work is complete, stop the workers. This releases the processes and their memory.
daemons(0)If you do not call daemons() first, in_parallel() has no workers to use. In that case map() runs each call in your own session, one at a time. This is nice because it means the same code can run in parallel or serially (e.g. on different machines).