Example 10: Parallel iteration with in_parallel()

Load our packages

library(here)      # manage file paths
library(socviz)    # data and some useful functions
library(tidyverse) # your friend and mine
library(mirai)     # parallel workers for purrr

A slow task

A bootstrap draws many random samples from the data, with replacement. It then calculates the same statistic on each sample. Each sample is independent of the others, so the task is easy to divide among several processes.

This function draws one bootstrap sample from organdata. It fits a model of the donor rate on consent law and returns the coefficient for presumed consent. The Sys.sleep() call adds a quarter of a second, to make the function slow.

boot_coef <- function(i, data) {
  boot_sample <- dplyr::slice_sample(data, prop = 1, replace = TRUE)
  fit <- stats::lm(donors ~ consent_law, data = boot_sample)
  Sys.sleep(0.25)
  tibble::tibble(rep = i, estimate = stats::coef(fit)[["consent_lawPresumed"]])
}

boot_coef(1, organdata)
# A tibble: 1 × 2
    rep estimate
  <dbl>    <dbl>
1     1     3.84

The function refers to package functions with ::, for example dplyr::slice_sample(). The parallel workers do not automatically load our packages. The :: prefix tells each worker where to find the function.

Run it sequentially

First, we run 24 replicates with an ordinary map(). Each replicate takes at least 0.25 seconds, so this takes at least six seconds.

system.time(
  seq_out <- map(1:24, \(i) boot_coef(i, organdata)) |>
    list_rbind()
)
   user  system elapsed 
  0.100   0.002   6.178 

Start the workers

daemons() from the mirai package starts a set of background R processes. These processes are the workers. Do not start more workers than the number of cores in your computer. Here we use four workers, or fewer if the computer has fewer cores.

The seed argument makes the random numbers in the workers reproducible.

n_workers <- min(4, parallel::detectCores() - 1)

daemons(n_workers, seed = 1234)

status()$connections
[1] 4

Which process does the work?

Sys.getpid() returns the ID number of the R process that runs it. In an ordinary map(), each call runs in our own R session. So each call returns the same number.

map_int(1:8, \(x) Sys.getpid()) |>
  unique()
[1] 6897

When we put the function inside in_parallel(), purrr sends each call to a worker. Each call can return a different number.

map_int(1:8, in_parallel(\(x) Sys.getpid())) |>
  unique()
[1] 6933 6948 6935 6940

Run it in parallel

Next, we run the bootstrap with in_parallel(). The workers do not see the objects in our session. So we must give them each object that the function needs. Here, the function needs boot_coef() and organdata. We supply them as named arguments to in_parallel().

system.time(
  par_out <- map(
    1:24,
    in_parallel(
      \(i) boot_coef(i, organdata),
      boot_coef = boot_coef,
      organdata = organdata
    )
  ) |>
    list_rbind()
)
   user  system elapsed 
  0.004   0.007   1.667 

The time is approximately the sequential time divided by the number of workers. There’s some overhead. Each worker must also start and receive the data, so the gain is a little smaller than that.

A common error

If we do not supply an object, the workers cannot find it. The map stops with an error.

map(1:4, in_parallel(\(i) boot_coef(i, organdata)))
Error in `map()`:
ℹ In index: 1.
Caused by error in `boot_coef()`:
! could not find function "boot_coef"

Look at the results

The parallel version gives one row per replicate, the same as the sequential version. The random samples are different, so the estimates are different. They have approximately the same distribution.

bind_rows(
  Sequential = seq_out,
  Parallel = par_out,
  .id = "method"
) |>
  summarize(
    mean = mean(estimate),
    sd = sd(estimate),
    .by = method
  )
# A tibble: 2 × 3
  method      mean    sd
  <chr>      <dbl> <dbl>
1 Sequential  3.24 0.646
2 Parallel    3.32 0.730

Stop the workers

When the parallel work is complete, stop the workers. This releases the processes and their memory.

daemons(0)

If you do not call daemons() first, in_parallel() has no workers to use. In that case map() runs each call in your own session, one at a time. This is nice because it means the same code can run in parallel or serially (e.g. on different machines).