Cloudbuild+playground+infra

cloudbuild.yaml

@@ -0,0 +1,58 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+steps:
+  - name: 'ubuntu'
+    entrypoint: 'bash'
+    args:
+      - '-c'
+      - |
+        apt-get update
+        export DEBIAN_FRONTEND=noninteractive
+        apt-get install -y software-properties-common
+        add-apt-repository -y ppa:deadsnakes/ppa && apt update
+        apt install -y python3.8 python3-pip
+        cd playground/infrastructure
+        pip install -r requirements.txt
+        export \
+        BEAM_ROOT_DIR="../../" \
+        SDK_CONFIG="../../playground/sdks.yaml" \
+        BEAM_EXAMPLE_CATEGORIES="../categories.yaml" \
+        SERVER_ADDRESS="https://${_SDK}.playground-uat.xyz" \
+        BEAM_USE_WEBGRPC=yes \
+        BEAM_CONCURRENCY=4
+        python3 ci_cd.py --step ${_STEP} --sdk SDK_${_SDK} --origin ${_ORIGIN} \
+        --subdirs ${_SUBDIRS}
+
+    env:
+      - 'ORIGIN=${_ORIGIN}'
+      - 'SDK=${_SDK}'
+      - 'STEP=${_STEP}'
+      - 'SUBDIRS=${_SUBDIRS}'
+      - 'GOOGLE_CLOUD_PROJECT=$PROJECT_ID'
+
+substitutions:
+  _STEP: CD
+  _SDK: GO
+  _ORIGIN: PG_EXAMPLES
+  _SUBDIRS: "../../learning/katas ../../examples ../../sdks"
+
+# This option enables writing logs to Cloud Logging
+options:
+  logging: CLOUD_LOGGING_ONLY
+
+timeout: 3600s

cloudbuild/configs/playground-infrastructure/README.md

@@ -0,0 +1,53 @@
+
+# Setup of Cloud Build and GitHub repository to deploy Beam Playground
+
+**This README explains how to:**
+
+1. Enable Cloud Build in your GCP Project
+
+2. Connect Github repository to Cloud Build
+
+
+### Before you begin
+
+- Create GCP project
+    - [Create a project docs](https://cloud.google.com/apis/docs/getting-started#creating_a_google_project)
+- Enable billing for the project
+    - [Enable billing docs](https://cloud.google.com/apis/docs/getting-started#enabling_billing)
+- Enable Cloud Build API
+    - [Enable API docs](https://cloud.google.com/apis/docs/getting-started#enabling_apis)
+- Have your GitHub repository ready
+
+## Enable Cloud Build and connect a GitHub Repository:
+
+To connect your GitHub repository to your Cloud Build:
+
+1. Open the Cloud Build, either from Navigation Menu on the left side or using search engine.
+
+2. Open the Triggers page in the Cloud Build console. Select required GCP project in the top bar.
+
+3. Click Connect repository on the top. Select GitHub (Cloud Build GitHub App), check the consent checkbox, and click Continue.
+
+4. In the pop-up that appears, select your GitHub username or organization.
+
+5. Select one of the following options based on your business need:
+
+        a. All repositories - enable current and future GitHub repositories for access via the Cloud Build app
+
+        b. Only select repositories - use the Select repositories drop-down to enable only specific repositories for access via the Cloud Build app.
+
+6. Click Install to install the Cloud Build app.
+
+7. After you have selected a project or created a new one, you will see the Connect repository panel.
+
+8. In the Select repository section, select the following fields:
+
+* GitHub account: The GitHub account used to install the Cloud Build GitHub App. This field may be pre-selected for you.
+
+* Repository: The repositories you want to connect to Cloud Build.
+
+9. Once you have selected your GitHub account and repositories, read the consent disclaimer and select the checkbox next to proceed further.
+
+10. Click Connect and Done.
+
+You have now connected one or more GitHub repositories to your Cloud project and Cloud Build. 

cloudbuild/configs/playground-infrastructure/cloudbuild.yaml

@@ -0,0 +1,66 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+steps:
+  # This step builds the base image for Beam Playground worker
+  - name: 'gcr.io/cloud-builders/docker'
+    args:
+      - 'build'
+      - '--tag=$_REGION-docker.pkg.dev/$PROJECT_ID/$_ARTIFACT_REGISTRY_REPOSITORY_ID/backend-builder:worker'
+      - '-f'
+      - 'cloudbuild/dockerfiles/Dockerfile-base-image-worker'
+      - '.'
+
+  # This step builds worker container and runs precommit gradle task
+  - name: '$_REGION-docker.pkg.dev/$PROJECT_ID/$_ARTIFACT_REGISTRY_REPOSITORY_ID/backend-builder:worker'
+    entrypoint: "/bin/bash"
+    args: ['-c', './gradlew playground:terraform:InitInfrastructure -Pproject_environment="beta"']
+
+  - name: '$_REGION-docker.pkg.dev/$PROJECT_ID/$_ARTIFACT_REGISTRY_REPOSITORY_ID/backend-builder:worker'
+    entrypoint: "gcloud"
+    args:
+      [
+        'container',
+        'clusters',
+        'get-credentials',
+        'playground-backend',
+        '--region=$_REGION'
+      ]
+
+  - name: '$_REGION-docker.pkg.dev/$PROJECT_ID/$_ARTIFACT_REGISTRY_REPOSITORY_ID/backend-builder:worker'
+    entrypoint: "gcloud"
+    args:
+      [
+        'app',
+        'deploy',
+        'playground/index.yaml'
+      ]
+
+  # This step builds worker container and runs precommit gradle task
+  - name: '$_REGION-docker.pkg.dev/$PROJECT_ID/$_ARTIFACT_REGISTRY_REPOSITORY_ID/backend-builder:worker'
+    entrypoint: "/bin/bash"
+    args: [ '-c', './gradlew playground:terraform:gkebackend -Pproject_environment="beta" -Pdocker-tag="ruslan"' ]
+
+substitutions:
+  _REGION: us-central1
+# This will need to be changed to reflect your actual Artifact Repository name for docker containers in GCP
+  _ARTIFACT_REGISTRY_REPOSITORY_ID: playground-repository
+
+images:
+  - '$_REGION-docker.pkg.dev/$PROJECT_ID/$_ARTIFACT_REGISTRY_REPOSITORY_ID/backend-builder:worker'
+
+timeout: 2000s

cloudbuild/dockerfiles/Dockerfile-base-image-worker

@@ -0,0 +1,56 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+FROM gcr.io/google.com/cloudsdktool/google-cloud-cli
+
+ARG GO_VERSION=1.18
+ENV PATH="${PATH}:/usr/local/go/bin"
+
+RUN apt-get update >/dev/null
+RUN apt-get -y install build-essential unzip
+
+# Install Docker
+# See: https://docs.docker.com/engine/install/ubuntu/#install-using-the-convenience-script
+RUN curl -fsSL https://get.docker.com -o get-docker.sh
+RUN sh ./get-docker.sh >/dev/null
+
+# Install Go
+# See: https://go.dev/doc/install
+RUN curl -OL https://golang.org/dl/go$GO_VERSION.linux-amd64.tar.gz
+RUN sha256sum go$GO_VERSION.linux-amd64.tar.gz
+RUN tar -C /usr/local -xvf go$GO_VERSION.linux-amd64.tar.gz
+RUN rm go$GO_VERSION.linux-amd64.tar.gz
+RUN go version
+
+#Install kubectl
+RUN curl -LO "https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl" \
+    && chmod +x ./kubectl \
+    && mv ./kubectl /usr/local/bin/kubectl \
+    && curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 \
+    && chmod +x get_helm.sh && ./get_helm.sh > /dev/null
+
+#Install Terraform
+RUN curl -OL https://releases.hashicorp.com/terraform/1.3.3/terraform_1.3.3_linux_amd64.zip
+RUN unzip terraform_1.3.3_linux_amd64.zip
+RUN mv terraform /usr/local/bin
+
+#Install GoLint
+RUN curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s v1.50.1
+
+# Install Java
+RUN apt-get install openjdk-11-jdk -y >/dev/null
+RUN java -version

cloudbuild_cd_python.yaml

@@ -0,0 +1,61 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+steps:
+  - name: 'ubuntu'
+    entrypoint: 'bash'
+    args:
+      - '-c'
+      - |
+        apt-get update
+        export DEBIAN_FRONTEND=noninteractive
+        apt-get install -y software-properties-common
+        add-apt-repository -y ppa:deadsnakes/ppa && apt update
+        apt install -y python3.8 python3-pip
+        cd playground/infrastructure
+        pip install -r requirements.txt
+        export \
+        BEAM_ROOT_DIR="../../" \
+        SDK_CONFIG="../../playground/sdks.yaml" \
+        BEAM_EXAMPLE_CATEGORIES="../categories.yaml" \
+        SERVER_ADDRESS="https://${_SDK}.playground-uat.xyz" \
+        BEAM_USE_WEBGRPC=yes \
+        BEAM_CONCURRENCY=4
+        for sdk in go java python
+        do
+            export sdks=$sdk
+            export SERVER_ADDRESS=https://$sdks-playground 
+            python3 ci_cd.py --step ${_STEP} --sdk SDK_$sdks --origin ${_ORIGIN} --subdirs ${_SUBDIRS}
+        done
+
+    env:
+      - 'ORIGIN=${_ORIGIN}'
+      - 'STEP=${_STEP}'
+      - 'SUBDIRS=${_SUBDIRS}'
+      - 'GOOGLE_CLOUD_PROJECT=$PROJECT_ID'
+
+substitutions:
+  _STEP: CD
+  _SDK: JAVA
+  _ORIGIN: PG_EXAMPLES
+  _SUBDIRS: "././learning/katas ././examples ././sdks"
+
+# This option enables writing logs to Cloud Logging
+options:
+  logging: CLOUD_LOGGING_ONLY
+
+timeout: 3600s

learning/tour-of-beam/terraform-v2/README.md

@@ -0,0 +1,79 @@
+<!--
+    Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+      http://www.apache.org/licenses/LICENSE-2.0
+    Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+-->
+
+# Overview
+
+This project deploys backend infrastructure for Tour of Beam as GCP Cloud Functions using Terraform.
+
+# Requirements
+
+## Development requirements
+
+- [Active GCP project with billing enabled](https://developers.google.com/workspace/guides/create-project)
+- [Existing bucket to store Terraform state](https://cloud.google.com/storage/docs/creating-buckets)
+- [Existing service account with the following roles](https://cloud.google.com/iam/docs/creating-managing-service-accounts):
+    - Cloud Functions Admin
+    - Project IAM Admin
+    - Service Account Admin
+    - Service Account User
+    - Storage Admin
+    - Storage Object Admin
+
+#### Configuring your environment
+
+Steps below will:
+1. Enable required APIs for the project
+2. Create service account and assign required IAM roles to it (service account to run the function with)
+3. Create bucket to archive and store source code
+4. Create cloud functions to each function defined in backend source code
+
+
+```bash
+# Create environment directory per your requirements/policy
+mkdir environments/dev
+cd ../environments/dev
+# Create new configuration to auth to GCP Project
+gcloud init
+# Acquire new user credentials to use for Application Default Credentials
+gcloud auth application-default login
+# Initiliaze and run terraform
+terraform init
+terraform plan
+terraform apply
+```
+
+
+### Sample usage
+
+Entry point: list sdk names
+```
+$ curl -X GET https://$REGION-$PROJECT_ID.cloudfunctions.net/getSdkList | json_pp
+```
+[response](./samples/api/get_sdk_list.json)
+
+Get content tree by sdk name (SDK name == SDK id)
+```
+$ curl -X GET 'https://$REGION-$PROJECT_ID.cloudfunctions.net/getContentTree?sdk=python'
+```
+[response](./samples/api/get_content_tree.json)
+
+
+Get unit content tree by sdk name and unitId
+```
+$ curl -X GET 'https://$REGION-$PROJECT_ID.cloudfunctions.net/getContentTree?sdk=python&id=challenge1'
+```
+[response](./samples/api/get_unit_content.json)