Infrastructure-as-Code deploys API & dependencies (#29155)

.gitignore

@@ -143,3 +143,6 @@ playground/cloudfunction.zip
 
 # Exception to .gitignore .test-infra/pipelines related files
 !.test-infra/pipelines/**/apache-beam-testing.tfvars
+
+# Ignore .test-infra/mock-apis related files
+.test-infra/mock-apis/**/charts/

.test-infra/mock-apis/README.md

@@ -113,4 +113,97 @@ Follow these steps to run the services on your local machine.
 
 # Deployment
 
-TODO: See https://github.com/apache/beam/issues/28709
+The following has already been performed for the `apache-beam-testing` project
+and only needs to be done for a different Google Cloud project.
+
+To deploy the APIs and dependent services, run the following commands.
+
+## 1. Provision dependent resources in Google Cloud.
+
+```
+terraform -chdir=infrastructure/terraform init
+terraform -chdir=infrastructure/terraform apply -var-file=apache-beam-testing.tfvars
+```
+
+## 2. Set the KO_DOCKER_REPO environment variable.
+
+After the terraform module completes, you will need to set the following:
+
+```
+export KO_DOCKER_REPO=<region>-docker.pkg.dev/<project>/<repository>
+```
+
+where:
+
+- `region` - is the GCP compute region
+- `project` - is the GCP project id i.e. `apache-beam-testing`
+- `repository` - is the repository name created by the terraform module. To
+find this run:
+`gcloud artifacts repositories list --project=<project> --location=<region>`.
+For example,
+`gcloud artifacts repositories list --project=apache-beam-testing --location=us-west1`
+
+## 3. Connect to the Kubernetes cluster
+
+Run the following command to setup credentials to the Kubernetes cluster.
+
+```
+gcloud container clusters get-credentials <cluster> --region <region> --project <project>
+```
+
+where:
+- `region` - is the GCP compute region
+- `project` - is the GCP project id i.e. `apache-beam-testing`
+- `<cluster>` - is the name of the cluster created by the terraform module.
+You can find this by running `gcloud container clusters list --project=<project> --region=<region>`
+
+## 4. Provision the Redis instance
+
+```
+kubectl kustomize --enable-helm infrastructure/kubernetes/redis | kubectl apply -f -
+```
+
+**You will initially see "Unschedulable" while the cluster is applying the helm
+chart. It's important to wait until the helm chart completely provisions resources
+before proceeding. Using Google Kubernetes Engine (GKE) autopilot may take some
+time before this autoscales appropriately. **
+
+## 5. Provision the Echo service
+
+Run the following command to provision the Echo service.
+
+```
+kubectl kustomize infrastructure/kubernetes/echo | ko resolve -f - | kubectl apply -f -
+```
+
+Like previously, you may see "Does not have minimum availability" message
+showing on the status. It may take some time for GKE autopilot
+to scale the node pool.
+
+## 6. Provision the Refresher services
+
+The Refresher service relies on [kustomize](https://kustomize.io) overlays
+which are located at [infrastructure/kubernetes/refresher/overlays](infrastructure/kubernetes/refresher/overlays).
+
+Each folder contained in [infrastructure/kubernetes/refresher/overlays](infrastructure/kubernetes/refresher/overlays)
+corresponds to an individual Refresher instance that is identified by the UUID.
+You will need to deploy each one individually.
+
+For example:
+```
+kubectl kustomize infrastructure/kubernetes/refresher/overlays/f588787b-28f8-4e5f-8335-f862379daf59 | ko resolve -f - | kubectl apply -f -
+```
+
+Like previously, you may see "Does not have minimum availability" message
+showing on the status. It may take some time for GKE autopilot
+to scale the node pool.
+
+## Additional note for creating a new Refresher service instance
+
+Each Refresher service instance relies on a unique UUID, where
+the [kustomize](https://kustomize.io) overlay replaces in the
+[infrastructure/kubernetes/refresher/base](infrastructure/kubernetes/refresher/base)
+template.
+
+You can copy the entire folder and paste into a new one with a unique UUID
+and then perform a find-replace of the old UUID with the new one.

.test-infra/mock-apis/infrastructure/kubernetes/echo/configmap.yaml

@@ -0,0 +1,30 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Configures the Echo Service ConfigMap.
+# See https://github.com/apache/beam/blob/master/.test-infra/mock-apis/src/main/go/cmd/service/echo/main.go
+# for details on the Echo service executable and
+# https://github.com/apache/beam/blob/master/.test-infra/mock-apis/src/main/go/internal/environment/variable.go
+# for details on various environment variables.
+
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: echo
+data:
+  HTTP_PORT: "8080"
+  GRPC_PORT: "50051"
+  # See .test-infra/mock-apis/infrastructure/kubernetes/redis
+  CACHE_HOST: redis-master.default.svc.cluster.local:6379

.test-infra/mock-apis/infrastructure/kubernetes/echo/deployment.yaml

@@ -0,0 +1,48 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Configures the Echo Deployment on the Kubernetes cluster.
+# See https://github.com/apache/beam/blob/master/.test-infra/mock-apis/src/main/go/cmd/service/echo/main.go
+# for details on the Echo service executable.
+# Assumes usage of https://ko.build/ to resolve the manifest:
+# export KO_DOCKER_REPO=<location>-docker.pkg.dev/<project>/<repository>
+# kubectl kustomize .test-infra/mock-apis/infrastructure/echo | ko resolve -f - | kubectl apply -f -
+# See .test-infra/mock-apis/README.md for details
+
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: echo
+  labels:
+    app: echo
+spec:
+  replicas: 3
+  selector:
+    matchLabels:
+      app: echo
+  template:
+    metadata:
+      labels:
+        app: echo
+    spec:
+      containers:
+        - name: echo
+          # Prefixed with ko:// to resolve with the ko utility.
+          # See https://ko.build/features/k8s/ for details.
+          image: ko://github.com/apache/beam/test-infra/mock-apis/src/main/go/cmd/service/echo
+          imagePullPolicy: IfNotPresent
+          envFrom:
+            - configMapRef:
+                name: echo

.test-infra/mock-apis/infrastructure/kubernetes/echo/kustomization.yaml

@@ -0,0 +1,23 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Configures the provisioning of the Echo service on the Kubernetes cluster.
+# See https://github.com/apache/beam/blob/master/.test-infra/mock-apis/src/main/go/cmd/service/echo/main.go
+# for details on the Echo service executable.
+
+resources:
+- configmap.yaml
+- deployment.yaml
+- service.yaml

.test-infra/mock-apis/infrastructure/kubernetes/echo/service.yaml

@@ -0,0 +1,42 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Configures the provisioning of the Echo Service on the Kubernetes cluster.
+# See https://github.com/apache/beam/blob/master/.test-infra/mock-apis/src/main/go/cmd/service/echo/main.go
+# for details on the Echo service executable.
+
+apiVersion: v1
+kind: Service
+metadata:
+  name: echo
+  annotations:
+    # Configures the LoadBalancer to assign an internal private IP
+    # instead of an external private IP.
+    # See https://cloud.google.com/kubernetes-engine/docs/how-to/internal-load-balancing
+    networking.gke.io/load-balancer-type: "Internal"
+spec:
+  type: LoadBalancer
+  externalTrafficPolicy: Cluster
+  selector:
+    app: echo
+  # Ports must match the environment variables assigned in the ConfigMap/echo.
+  # See configmap.yaml.
+  ports:
+  - port: 50051
+    name: grpc
+    targetPort: 50051
+  - port: 8080
+    name: http
+    targetPort: 8080

.test-infra/mock-apis/infrastructure/kubernetes/redis/kustomization.yaml

@@ -0,0 +1,22 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Configures a Redis instance using https://bitnami.com/stack/redis/helm.
+helmCharts:
+  - name: redis
+    releaseName: redis
+    repo: https://charts.bitnami.com/bitnami
+    version: 18.1.5
+    valuesFile: redis-values.yaml

.test-infra/mock-apis/infrastructure/kubernetes/redis/redis-values.yaml

@@ -0,0 +1,26 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Assigns values to the https://bitnami.com/stack/redis/helm chart.
+auth:
+  # The cluster is used for testing only in a private Google Kubernetes Engine
+  # (GKE) cluster. So setting enabled to false delegates role based
+  # access control to Google Cloud Identity and Access Management (IAM).
+  enabled: false
+
+  # We set sentinel to false, since we do not need high availability.
+  # See https://developer.redis.com/operate/redis-at-scale/high-availability/understanding-sentinels/
+  # for more details on the sentinel mode.
+  sentinel: false

.test-infra/mock-apis/infrastructure/kubernetes/refresher/base/configmap.yaml

@@ -0,0 +1,37 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Configures the Refresher ConfigMap.
+# See https://github.com/apache/beam/blob/master/.test-infra/mock-apis/src/main/go/cmd/service/refresher/main.go
+# Designed for use with kustomize patch overlays.
+# See https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/patches/
+
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: refresher
+  labels:
+    app.kubernetes.io/name: refresher
+
+    # targeted for overlay replacement
+    quota-id: quota-id-value
+data:
+  CACHE_HOST: redis-master.default.svc.cluster.local:6379
+
+  # targeted for overlay replacement
+  QUOTA_ID: quota-id-value
+  QUOTA_SIZE: "100"
+  QUOTA_REFRESH_INTERVAL: "10s"
+

.test-infra/mock-apis/infrastructure/kubernetes/refresher/base/deployment.yaml

@@ -0,0 +1,54 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Configures the Refresher Deployment.
+# See https://github.com/apache/beam/blob/master/.test-infra/mock-apis/src/main/go/cmd/service/refresher/main.go
+# Designed for use with kustomize patch overlays.
+# See https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/patches/
+
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  # name created using kustomize nameSuffix as refresher-<quota-id-value>
+  name: refresher
+
+  labels:
+    app.kubernetes.io/name: refresher
+
+    # targeted for overlay replacement
+    quota-id: quota-id-value
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app.kubernetes.io/name: refresher
+
+      # targeted for overlay replacement
+      quota-id: quota-id-value
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/name: refresher
+
+        # targeted for overlay replacement
+        quota-id: quota-id-value
+    spec:
+      containers:
+        - name: refresher
+          image: ko://github.com/apache/beam/test-infra/mock-apis/src/main/go/cmd/service/refresher
+          imagePullPolicy: IfNotPresent
+          envFrom:
+            - configMapRef:
+                name: refresher