everybody
As the title says, I would be curious to understand why I can't connect via Sparklyr to gCloud clusters using kubernetes.
The steps to configure the system were as follows:
- project creation on gCloud (free version 300dollars)
- Cloud SDK installation on MacOS
- Installation Kubectl binary with curl on MacOS
- Docker installation
- By terminal:
- configure the cluster
gcloud config set compute/zone us-central1-f gcloud container clusters create spark-on-gke --machine-type n1-standard-2 - bind the cluster admin to email
kubectl create clusterrolebinding user-admin-binding --clusterrole=cluster-admin --user=pesca@gmail.com kubectl create clusterrolebinding --clusterrole=cluster-admin --serviceaccount=default:default spark-admin - From R connect to the MASTER_IP, importing the public image offered by jluraschi
remotes::install_github("rstudio/sparklyr”); library(sparklyr) sc <- spark_connect(config = spark_config_kubernetes( "k8s://https://<k8s-ip>", account = "default", image = "docker.io/jluraschi/spark:sparklyr", version = "2.4"))
And the error that appears is: Error from server (NotFound): pods "sparklyr-c27317e4b89" not found