I am currently using furrr
to create a more organized execution of my model. I use a data.frame
to pass parameters to a function in a orderly way, and then using the furrr::future_map()
to map a function across all the parameters. The function works flawlessly when using the sequential and multicore futures on my local Machine (OSX).
Now, I want to test my code creating my own cluster of AWS instances (just as shown here).
I created a function using the linked article code:
make_cluster_ec2 <- function(public_ip){
ssh_private_key_file <- Sys.getenv('PEM_PATH')
github_pac <- Sys.getenv('PAC')
cl_multi <- future::makeClusterPSOCK(
workers = public_ip,
user = "ubuntu",
rshopts = c(
"-o", "StrictHostKeyChecking=no",
"-o", "IdentitiesOnly=yes",
"-i", ssh_private_key_file
),
rscript_args = c(
"-e", shQuote("local({p <- Sys.getenv('R_LIBS_USER'); dir.create(p, recursive = TRUE, showWarnings = FALSE); .libPaths(p)})"),
"-e", shQuote("install.packages('devtools')"),
"-e", shQuote(glue::glue("devtools::install_github('user/repo', auth_token = '{github_pac}')"))
),
dryrun = FALSE)
return(cl_multi)
}
Then, I create the cluster object and then check that is connected to the right instance
public_ids <- c('public_ip_1', public_ip_2')
cls <- make_cluster_ec2(public_ids)
f <- future(Sys.info())
And when I print f
I get the specs of one of my remote instances, which indicates the socket is correctly connected:
> value(f)
sysname
"Linux"
release
"4.15.0-1037-aws"
version
"#39-Ubuntu SMP Tue Apr 16 08:09:09 UTC 2019"
nodename
"ip-xxx-xx-xx-xxx"
machine
"x86_64"
login
"ubuntu"
user
"ubuntu"
effective_user
"ubuntu"
But when I run my code using my cluster plan:
plan(list(tweak(cluster, workers = cls), multisession))
parameter_df %>%
mutate(model_traj = furrr::future_pmap(list('lat' = latitude,
'lon' = longitude,
'height' = stack_height,
'name_source' = facility_name,
'id_source' = facility_id,
'duration' = duration,
'days' = seq_dates,
'daily_hours' = daily_hours,
'direction' = 'forward',
'met_type' = 'reanalysis',
'met_dir' = here::here('met'),
'exec_dir' = here::here("Hysplit4/exec"),
'cred'= list(creds)),
dirtywind::hysplit_trajectory,
.progress = TRUE)
)
I get the following error:
Error in file(temp_file, "a") : cannot open the connection
In addition: Warning message:
In file(temp_file, "a") :
cannot open file '/var/folders/rc/rbmg32js2qlf4d7cd4ts6x6h0000gn/T//RtmpPvdbV3/filecf23390c093.txt': No such file or directory
I can not figure out what is happening under the hood, and I can not traceback()
the error either from my remote machines. I have test the connection with the examples in the article and things seem to run correctly. I am wondering why is trying to create a tempdir
during the execution. What am I missing here?
(This is also an issue in the furrr
repo)