I'd like to set up an AWS architecture where I have a memory-optimized instance (say, an X1) running a head node for a dynamically allocated cluster, and then the rest of the cluster is spun on demand and on the cheap (e.g. via spot bids and on cpu- or even gpu-optimized instances). Moreover, I'd like that cluster to be transparent for use with R, either via foreach() or via bioconductor's parLapply() (MPI interface). The 1-hr spot bid requirement is probably fine for now. Could someone point me towards the relevant AWS mechanisms that don't involve submitting batch jobs using cli and instead support programmatic (R api) allocation of instances and subsequent distribution via the aforementioned R mechanisms? I don't think AWS parallel cluster is it, and I'm having a devil of a time finding the right mechanism.
↧