This repository contains all the required configuration and documentation to deploy the OpenEO service on Kubernetes on a DIAS cloud environment. The CloudFerro CreoDIAS Openstack was used for this setup, but should work for any Openstack environment.
The goal is to have a working OpenEO deployment, running as a Spark job, on Kubernetes, on Openstack.
- CreoDIAS account with credits to provision Openstack resources
- Some Kubernetes knowledge
- A working Kubernetes cluster (see Setup the Kubernetes environment)
To deploy a Kubernetes cluster on the CreoDIAS OpenStack, we make use of RKE, Rancher Kubernetes Engine. This tool allows you to deploy a cluster that runs in containers in a fast and reliable way. Rancher also provides a Terraform provider for RKE, which we use to keep all of our infrastructure provisioning within Terraform.
Steps we took to create a cluster:
- Create an OpenStack image with Packer with all necessary dependencies like Docker, users, ...
- Provision instances, networks, security groups, ...
- Provision the Kubernetes cluster itself
Since Spark version 2.3.0, you can use Kubernetes as scheduler for your Spark jobs (docs). Using this functionality, you just use the regular spark-submit tool, with Kubernetes specific parameters. While this would work, it has some drawbacks:
- You can't manage your Spark applications as regular Kubernetes objects and thus not use
kubectlto manage them - There is no way of native cron support
- No automatic retries
- No built-in monitoring
To meet all these shortcomings, the GoogleCloudPlatform has developed the spark-on-k8s-operator. This operator provides a way to schedule Spark applications as native Kubernetes resources, using CRD's (Custom Resource Definitions). The operator will also manage the lifecycle of the Spark applications and provide many other features (see the Github repo for a complete feature list).
Now that we've chosen to use the operator instead of regular spark-submit commands, we have to get the operator installed on the cluster. The easiest way to perform the installation, is by using the Helm chart. Installation instructions on how to install Helm can be found here.
With Helm installed, we can now install the operator. To start, we need to add the Helm chart repository to Helm:
helm repo add spark-operator https://googlecloudplatform.github.io/spark-on-k8s-operator
You can choose in which Kubernetes namespace the spark operator will be installed. In this guide. we'll create a separate namespace for the operator and one for the Spark jobs. The namespace for the Spark jobs should be created separately, as the Helm chart doesn't create it for us.
kubectl create namespace spark-jobs
helm install sparkoperator --generate-name --create-namespace --namespace spark-operator --set sparkJobNamespace=spark-jobs --set webhook.enable=true --set image.tag=v1beta2-1.3.0-3.1.1
Let's break down the different options passed to the helm install command:
| Option | Explanation |
|---|---|
| incubator/sparkoperator | Helm chart that will be installed |
| --generate-name | Generate a name for the Helm release (Also possible to provide your own name) |
| --create-namespace | Create the namespace if it doesn't exist yet (requires Helm 3.2+) |
| --namespace | The namespace where the operator will be installed in |
| --set sparkJobNamespace | The namespace where the Spark jobs will be deployed |
| --set webhook.enable | This enables the mutating admission webhook |
With the operator installed you should be able to get the following outputs:
helm list -n spark-operator
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
spark-operator spark-operator 1 2022-05-13 06:17:20.165279131 +0000 UTC deployed spark-operator-1.1.15 v1beta2-1.3.1-3.1.1
and:
kubectl get pods -n spark-operator
NAME READY STATUS RESTARTS AGE
sparkoperator-1593174963-556544cb66-v5v7f 1/1 Running 0 11m
The Spark operator is now up and running in its own namespace.
For data access, openEO requires you to register the configuration of 'collections' in a file inside the docker image. The default location is '/opt/layercatalog.json', but it can be set with an environment variable OPENEO_CATALOG_FILES.
The main format of this file follows the STAC collection metadata specification, but there's also a number of custom properties. Documentation of these properties is rather sparse, so for now, working from existing examples is the best approach. In the best case, when working from a properly configured STAC collection, the amount of additional configuration is limited.
To update the configuration in the docker file, we would recommend rebuilding the image. Another approach might be to try and use a Kubernetes config map.
An example of a layercatalog: https://github.com/Open-EO/openeo-geotrellis-kubernetes/blob/master/docker/creo_layercatalog.json
For reference, this Python code actually works with the configuration in the layer catalog: https://github.com/Open-EO/openeo-geopyspark-driver/blob/master/openeogeotrellis/layercatalog.py
Custom config for STAC collection. Note that 'opensearch_XX' properties are used, but the backend tries to determine automatically what to use.
"_vito": {
"data_source": {
"type": "file-s2",
"opensearch_collection_id": "S2",
"opensearch_endpoint": "https://resto.c-scale.zcu.cz",
"provider:backend": "incd"
}
}
Now that we have the Spark operator running, it's time to deploy our application on the cluster. As Kubernetes is a container orchestrator, we of course need to package our application into a container image. The necessary files to build this container image can be found in the docker directory. A prebuilt image is available at vito-docker.artifactory.vgt.vito.be/openeo-geotrellis-kube.
As we are using the Spark operator, we can now define our Spark job as a Kubernetes resource, rather than a spark-submit script.
To have a fully functional application, we need more than a SparkApplication Kubernetes resource. We also need an Ingress, ServiceAccounts, RBAC, ... A Helm chart was written to help with all the parts we need. Instructions on how to use this chart, can be found in the README.md file.
After creating a values.yam file with your necessary values, you can then invoke a regular helm install command to deploy your instance of openEO to your Kubernetes cluster.
The configuration of the backend is mostly done via environment variables in values.yaml. Please review these values for your backend:
| Service | Function |
|---|---|
| SWIFT_URL | url of S3 API that will be used to store batch job results |
| AWS_ACCESS_KEY_ID | S3 Access key used to manage batch job results |
| AWS_SECRET_ACCESS_KEY | S3 Access key secret |
| ZOOKEEPERNODES | Zookeeper cluster used for persistence |
To track the status of batch jobs, we need a job-tracker cron job. An example of such job can be found at examples/job-tracker.yaml.
The spark-operator also provides an easy way to monitor your Spark Applications that are submitted by the operator. In the openeo.yaml manifest, you can find the necessary configuration:
monitoring:
exposeDriverMetrics: true
exposeExecutorMetrics: true
prometheus:
jmxExporterJar: "/opt/jmx_prometheus_javaagent-0.13.0.jar"
port: 8090
You can also only expose the executor's metrics for example.
This monitoring section uses a default configuration file for the jmx_exporter. The default configuration file can be found here. Of course, there is also a possibility to override this configuration. Via a Kubernetes configMap, you can mount a different prometheus.yaml file in your driver and executor. First, a configMap should be created, containing the prometheus.yaml file:
** To make the configMaps to work, you need to enable the mutating admission webhook for the spark-operator **
kubectl create configmap prometheus-jmx-config --from-file=prometheus.yaml
This configMap can now be added to your pods:
driver:
configMaps:
- name: prometheus-jmx-config
path: /opt/prometheus_config
The path can then be used in the monitoring configuration:
monitoring:
prometheus:
configuration: /opt/prometheus_config
The new metrics should now be appearing in your Prometheus instance.
| Service | Function |
|---|---|
| Prometheus | Monitoring |
| Grafana | Visualize prometheus metrics |
| Alertmanager | Alerting |
| Spark History Server | Overview of historical jobs |
| Zookeeper | Keep track of batch jobs |
| Traefik | Ingress |
| Kubecost | Monitoring of costs per namespace, pod, ... |
| Filebeat | Log aggregation |
| RKE Pushprox | Helper for metrics of RKE components |
| Cinder CSI | Dynamic OpenStack Cinder volumes |