[RRIO] Define and implement mock quota aware API (#28893)

* Implement gRPC service with transcoder * WIP: defer integration tests for later * Make service executables * remove grpc gateway * Clean up and add messy integration test * Delete flakey integration tests that depend on Docker * Generate java proto * Generate python proto * Python is too much trouble for now * Finalize README * Fix rat and whitespace * Fix whitespace * Clean up per PR comments * Fix gcp logging not capturing severity
apache · Oct 23, 2023 · a07d90e · a07d90e
1 parent ca96cb5
commit a07d90e
Show file tree

Hide file tree

Showing 30 changed files with 4,324 additions and 0 deletions.
diff --git a/.test-infra/mock-apis/README.md b/.test-infra/mock-apis/README.md
@@ -0,0 +1,116 @@
+<!--
+    Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+    Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+-->
+
+# Overview
+
+This directory holds code and related artifacts to support API related
+integration tests.
+
+## System review
+
+The diagram below summarizes the system design. Integration tests use an API
+client that makes calls to a backend service. Prior to fulfilling the response,
+the service checks and decrements a quota. Said quota persists in a backend
+redis instance that is refreshed on an interval by the
+[Refresher](./src/main/go/cmd/service/refresher).
+
+## Echo Service
+
+The [Echo Service](./src/main/go/cmd/service/echo) implements a simple gRPC
+service that echos a payload. See [echo.proto](./proto/echo/v1/echo.proto)
+for details.
+
+```mermaid
+flowchart LR
+    echoClient --> echoSvc
+    subgraph "Integration Tests"
+       echoClient[Echo Client]
+    end
+    subgraph Backend
+       echoSvc[Echo Service&lt./src/main/go/cmd/service/echo&gt]
+       refresher[Refresher&lt./src/main/go/cmd/service/refresher&gt]
+       redis[redis://:6739]
+       refresher -- SetQuota&#40&ltstring&gt,&ltint64&gt,&lttime.Duration&gt&#41 --> redis
+       echoSvc -- DecrementQuota&#40&ltstring&gt&#41 --> redis
+    end
+```
+
+# Writing Integration Tests
+
+TODO: See https://github.com/apache/beam/issues/28859
+
+# Development Dependencies
+
+| Dependency                                          | Reason                                                                                 |
+|-----------------------------------------------------|----------------------------------------------------------------------------------------|
+| [go](https://go.dev)                                | For making code changes in this directory. See [go.mod](go.mod) for required version.  |
+| [buf](https://github.com/bufbuild/buf#installation) | Optional for when making changes to proto.                                             |
+| [ko](https://ko.build/install/)                     | To easily build Go container images.                                                   |
+
+# Testing
+
+## Unit
+
+To run unit tests in this project, execute the following command:
+
+```
+go test ./src/main/go/internal/...
+```
+
+## Integration
+
+TODO: See https://github.com/apache/beam/issues/28859
+
+# Local Usage
+
+## Requirements
+
+To execute the services on your local machine, you'll need [redis](https://redis.io/docs/getting-started/installation/).
+
+## Execute services
+
+Follow these steps to run the services on your local machine.
+
+
+1. Start redis
+
+    Start redis using the following command.
+    ```
+    redis-server
+    ```
+
+1. Start the refresher service in a new terminal.
+    ```
+    export CACHE_HOST=localhost:6379; \
+    export QUOTA_ID=$(uuidgen); \
+    export QUOTA_REFRESH_INTERVAL=10s; \
+    export QUOTA_SIZE=100; \
+    go run ./src/main/go/cmd/service/refresher
+    ```
+1. Start the echo service in a new terminal.
+    ```
+    export HTTP_PORT=8080; \
+    export GRPC_PORT=50051; \
+    export CACHE_HOST=localhost:6379; \
+    go run ./src/main/go/cmd/service/echo
+    ```
+
+# Deployment
+
+TODO: See https://github.com/apache/beam/issues/28709
diff --git a/.test-infra/mock-apis/buf.gen.yaml b/.test-infra/mock-apis/buf.gen.yaml
@@ -0,0 +1,40 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# buf.gen.yaml configures proto stub generation using buf.
+#
+# Requirements:
+# - go install google.golang.org/protobuf/cmd/protoc-gen-go@latest
+# - go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest
+# - https://github.com/grpc/grpc-java/blob/master/compiler/README.md#grpc-java-codegen-plugin-for-protobuf-compiler
+# - https://grpc.io/docs/languages/python/quickstart/#grpc-tools
+#
+# Usage:
+# Open a terminal in the same directory as this file and run:
+#
+# buf generate
+#
+# See https://buf.build/docs/ for more details.
+
+version: v1
+plugins:
+- name: go
+  out: src/main/go/internal
+- name: go-grpc
+  out: src/main/go/internal
+- name: java
+  out: src/main/java
+- name: grpc-java
+  out: src/main/java
diff --git a/.test-infra/mock-apis/buf.lock b/.test-infra/mock-apis/buf.lock
@@ -0,0 +1,7 @@
+# Generated by buf. DO NOT EDIT.
+version: v1
+deps:
+  - remote: buf.build
+    owner: googleapis
+    repository: googleapis
+    commit: 28151c0d0a1641bf938a7672c500e01d
diff --git a/.test-infra/mock-apis/buf.yaml b/.test-infra/mock-apis/buf.yaml
@@ -0,0 +1,20 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Configures buf to include various proto dependencies.
+# See buf.build for details.
+version: v1
+deps:
+- buf.build/googleapis/googleapis
diff --git a/.test-infra/mock-apis/build.gradle b/.test-infra/mock-apis/build.gradle
@@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+ plugins {
+     id 'org.apache.beam.module'
+ }
+
+ applyJavaNature(
+    exportJavadoc: false,
+    publish: false,
+ )
+
+description = "Apache Beam :: Test Infra :: Mock APIs"
+ext.summary = "Supports API related integration tests."
+
+def guavaVersion = "31.1-jre"
+def ioGrpcApiVersion = "1.53.0"
+def protobufVersion = "1.55.1"
+def protobufJavaVersion = "3.23.2"
+
+ dependencies {
+
+    // Required by autogenerated proto classes.
+    implementation "io.grpc:grpc-api:${ioGrpcApiVersion}"
+    implementation "com.google.guava:guava:${guavaVersion}"
+    implementation "io.grpc:grpc-protobuf:${protobufVersion}"
+    implementation "com.google.protobuf:protobuf-java:${protobufJavaVersion}"
+    implementation "io.grpc:grpc-stub:${protobufVersion}"
+ }
diff --git a/.test-infra/mock-apis/go.mod b/.test-infra/mock-apis/go.mod
@@ -0,0 +1,58 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements.  See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License.  You may obtain a copy of the License at
+//
+//    http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// This module contains all Go code used for Beam's SDKs. This file is placed
+// in this directory in order to cover the go code required for Java and Python
+// containers, as well as the entire Go SDK. Placing this file in the repository
+// root is not possible because it causes conflicts with a pre-existing vendor
+// directory.
+module github.com/apache/beam/test-infra/mock-apis
+
+go 1.21
+
+require (
+	cloud.google.com/go/logging v1.8.1
+	cloud.google.com/go/monitoring v1.16.0
+	github.com/google/go-cmp v0.5.9
+	github.com/redis/go-redis/v9 v9.2.1
+	google.golang.org/genproto/googleapis/api v0.0.0-20230822172742-b8732ec3820d
+	google.golang.org/grpc v1.58.2
+	google.golang.org/protobuf v1.31.0
+)
+
+require (
+	cloud.google.com/go v0.110.6 // indirect
+	cloud.google.com/go/compute v1.23.0 // indirect
+	cloud.google.com/go/compute/metadata v0.2.3 // indirect
+	cloud.google.com/go/longrunning v0.5.1 // indirect
+	github.com/cespare/xxhash/v2 v2.2.0 // indirect
+	github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect
+	github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
+	github.com/golang/protobuf v1.5.3 // indirect
+	github.com/google/s2a-go v0.1.4 // indirect
+	github.com/googleapis/enterprise-certificate-proxy v0.2.4 // indirect
+	github.com/googleapis/gax-go/v2 v2.12.0 // indirect
+	go.opencensus.io v0.24.0 // indirect
+	golang.org/x/crypto v0.13.0 // indirect
+	golang.org/x/net v0.15.0 // indirect
+	golang.org/x/oauth2 v0.12.0 // indirect
+	golang.org/x/sync v0.3.0 // indirect
+	golang.org/x/sys v0.12.0 // indirect
+	golang.org/x/text v0.13.0 // indirect
+	google.golang.org/api v0.128.0 // indirect
+	google.golang.org/appengine v1.6.7 // indirect
+	google.golang.org/genproto v0.0.0-20230803162519-f966b187b2e5 // indirect
+	google.golang.org/genproto/googleapis/rpc v0.0.0-20230822172742-b8732ec3820d // indirect
+)