feat: Introduce OpenTelemetry Metrics Recording #2500

lqiu96 · 2024-02-23T23:25:17Z

Builds off #2433, based on the design in go/java-gapic-otel-metrics-design.

Discovered two issues via showcase tests:

Otel's Method Name is different between gRPC and HttpJson #2502
HttpJson Implementation is recording Operation(Success|Fail) metric before retries are attempted #2503

These issues are not blocking for this PR.

…xporter

gax-java/gax/BUILD.bazel

gax-java/gax/src/test/java/com/google/api/gax/tracing/OpentelemetryMetricsRecorderTest.java

lqiu96 · 2024-02-27T21:23:43Z

gax-java/gax/src/main/java/com/google/api/gax/tracing/OpentelemetryMetricsRecorder.java

+  @Override
+  public void recordOperationLatency(double operationLatency, Map<String, String> attributes) {
+    operationLatencyRecorder.record(operationLatency, toOtelAttributes(attributes));
+  }
+
+  /**
+   * Record an operation made. The operation count number is stored in a LongCounter.
+   *
+   * <p>The operation count should always be 1 and this should be invoked once.
+   *
+   * @param count The number of operations made
+   * @param attributes Map of the attributes to store
+   */
+  @Override
+  public void recordOperationCount(long count, Map<String, String> attributes) {
+    operationCountRecorder.add(count, toOtelAttributes(attributes));
+  }


Both Operation record calls should only occur once. Do we want to validate this/ throw an exception if recordOperation(...) is invoked more than once?

I think we should rely on the existing framework. If there is a existing bug, it should not be the responsibility of the recorder to validate it as well, we should probably validate it in the MetricsTracer.

showcase/gapic-showcase/src/test/java/com/google/showcase/v1beta1/it/ITOtelMetrics.java

gax-java/gax/src/main/java/com/google/api/gax/tracing/OpentelemetryMetricsRecorder.java

blakeli0 · 2024-03-05T14:24:49Z

gax-java/gax/src/main/java/com/google/api/gax/tracing/OpentelemetryMetricsRecorder.java

+  @Override
+  public void recordOperationLatency(double operationLatency, Map<String, String> attributes) {
+    operationLatencyRecorder.record(operationLatency, toOtelAttributes(attributes));
+  }
+
+  /**
+   * Record an operation made. The operation count number is stored in a LongCounter.
+   *
+   * <p>The operation count should always be 1 and this should be invoked once.
+   *
+   * @param count The number of operations made
+   * @param attributes Map of the attributes to store
+   */
+  @Override
+  public void recordOperationCount(long count, Map<String, String> attributes) {
+    operationCountRecorder.add(count, toOtelAttributes(attributes));
+  }


I think we should rely on the existing framework. If there is a existing bug, it should not be the responsibility of the recorder to validate it as well, we should probably validate it in the MetricsTracer.

gax-java/pom.xml

showcase/pom.xml

…1+ times

gax-java/gax/src/main/java/com/google/api/gax/rpc/ClientSettings.java

gax-java/dependencies.properties

blakeli0 · 2024-03-06T03:24:06Z

gax-java/gax/src/main/java/com/google/api/gax/tracing/OpenTelemetryMetricsRecorder.java

+  public OpenTelemetryMetricsRecorder(OpenTelemetry openTelemetry, String serviceName) {
+    Meter meter =
+        openTelemetry
+            .meterBuilder("Gax-OtelMetrics")


Let's use gax-java. Based on the Javadoc, the value here should be the library name.

Sure, updated.

Co-authored-by: Blake Li <[email protected]>

gax-java/gax/src/main/java/com/google/api/gax/tracing/OpenTelemetryMetricsRecorder.java

…metryMetricsRecorder.java

gax-java/gax/src/main/java/com/google/api/gax/tracing/OpenTelemetryMetricsRecorder.java

…metryMetricsRecorder.java

sonarqubecloud · 2024-03-14T19:35:47Z

Quality Gate failed for 'gapic-generator-java-root'

Failed conditions
0.0% Coverage on New Code (required ≥ 80%)

See analysis details on SonarCloud

sonarqubecloud · 2024-03-14T19:39:00Z

Quality Gate passed for 'java_showcase_integration_tests'

Issues
3 New issues
0 Accepted issues

Measures
0 Security Hotspots
88.7% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

🤖 I have created a release *beep* *boop* --- <details><summary>2.38.0</summary> ## [2.38.0](v2.37.0...v2.38.0) (2024-03-15) ### Features * [common-protos] add `api_version` extension to `ServiceOptions`, for collaborative versioning ([d343be9](d343be9)) * [common-protos] add `api_version` extension to `ServiceOptions`, for collaborative versioning ([#2551](#2551)) ([d343be9](d343be9)) * add `ErrorReason.LOCATION_POLICY_VIOLATED` enum value ([d343be9](d343be9)) * add `ErrorReason.LOCATION_POLICY_VIOLATED` enum value ([d343be9](d343be9)) * add `Publishing.rest_reference_documentation_uri` to aid client library publication ([d343be9](d343be9)) * add `Publishing.rest_reference_documentation_uri` to aid client library publication ([d343be9](d343be9)) * Add shopping and chat common protos. ([#2553](#2553)) ([5f2d4e7](5f2d4e7)), closes [#2018](#2018) * get PR description from googleapis commits ([#2531](#2531)) ([c2ea697](c2ea697)) * Introduce OpenTelemetry Metrics Recording ([#2500](#2500)) ([b936580](b936580)) * skip build only commit ([#2555](#2555)) ([180c8a9](180c8a9)) * Universe Domain Environment Variable Support ([#2485](#2485)) ([1463d64](1463d64)) ### Dependencies * normalize dependencies ([#2574](#2574)) ([6622238](6622238)) * update arrow.version to v15.0.1 ([#2565](#2565)) ([b2c3f6a](b2c3f6a)) * update dependency com.fasterxml.jackson:jackson-bom to v2.17.0 ([#2564](#2564)) ([40ae7f9](40ae7f9)) * update dependency com.google.api-client:google-api-client-bom to v2.4.0 ([#2570](#2570)) ([f60441f](f60441f)) * update dependency com.google.errorprone:error_prone_annotations to v2.26.1 ([#2530](#2530)) ([7c1aaab](7c1aaab)) * update dependency com.google.errorprone:error_prone_annotations to v2.26.1 ([#2532](#2532)) ([447b4e1](447b4e1)) * update dependency io.netty:netty-tcnative-boringssl-static to v2.0.65.final ([#2547](#2547)) ([46e0e0f](46e0e0f)) * update dependency net.bytebuddy:byte-buddy to v1.14.12 ([#2522](#2522)) ([edfec32](edfec32)) * update google api dependencies ([#2484](#2484)) ([92e91bc](92e91bc)) * update google api dependencies ([#2538](#2538)) ([d9355cf](d9355cf)) * update googleapis/java-cloud-bom digest to 3f93d58 ([#2499](#2499)) ([5fd4d5e](5fd4d5e)) * update googleapis/java-cloud-bom digest to 659764f ([#2545](#2545)) ([d6c8be6](d6c8be6)) * update netty dependencies ([#2480](#2480)) ([40753c3](40753c3)) * update opentelemetry-java monorepo to v1.35.0 ([#2477](#2477)) ([3ecefff](3ecefff)) * update opentelemetry-java monorepo to v1.36.0 ([#2550](#2550)) ([9669c21](9669c21)) * update opentelemetry-java monorepo to v1.36.0 ([#2573](#2573)) ([f5f201e](f5f201e)) * update slf4j monorepo to v2.0.12 ([#2481](#2481)) ([363a354](363a354)) ### Documentation * minor tweaks to various comments ([d343be9](d343be9)) * minor tweaks to various comments ([d343be9](d343be9)) </details> --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>

product-auto-label bot added the size: xl Pull request size is extra large. label Feb 23, 2024

lqiu96 changed the title ~~feat: Introduce Metrics Recording via OpenTelemetry~~ feat: Introduce OpenTelemetry Metrics Recording Feb 26, 2024

product-auto-label bot added size: l Pull request size is large. and removed size: xl Pull request size is extra large. labels Feb 26, 2024

feat: Implement OpentelemetryMetricsRecorder

21407aa

lqiu96 force-pushed the implement-OpentelemetryMetricsRecorder branch from 50fa91a to 21407aa Compare February 26, 2024 17:04

product-auto-label bot added size: xl Pull request size is extra large. and removed size: l Pull request size is large. labels Feb 26, 2024

lqiu96 added 2 commits February 26, 2024 12:06

chore: Update CONTRIBUTING.md

f757811

chore: Fix lint issues

58fc09c

product-auto-label bot added size: l Pull request size is large. and removed size: xl Pull request size is extra large. labels Feb 26, 2024

This was referenced Feb 26, 2024

Otel's Method Name is different between gRPC and HttpJson #2502

Open

HttpJson Implementation is recording Operation(Success|Fail) metric before retries are attempted #2503

Closed

lqiu96 added 3 commits February 26, 2024 15:37

chore: Clean up ITOTelMetrics test

d29ad1a

chore: Clean up ITOTelMetrics test

df7c611

chore: Ignore failing Otel tests

9a3fc3c

product-auto-label bot added size: xl Pull request size is extra large. and removed size: l Pull request size is large. labels Feb 27, 2024

lqiu96 added 3 commits February 27, 2024 12:47

chore: Update OpentelemetryMetricsRecorderTest to use InMemoryMetricE…

9121cb6

…xporter

chore: Update showcase tests

9d099f6

chore: Add missing otel dependencies.properties values

03acef3

blakeli0 reviewed Feb 27, 2024

View reviewed changes

gax-java/gax/BUILD.bazel Show resolved Hide resolved

lqiu96 added 2 commits February 27, 2024 15:39

chore: Add javadocs for OpentelemetryMetricsRecorder

a2d979b

Merge branch 'main' into implement-OpentelemetryMetricsRecorder

d46fef7

lqiu96 commented Feb 27, 2024

View reviewed changes

gax-java/gax/src/test/java/com/google/api/gax/tracing/OpentelemetryMetricsRecorderTest.java Outdated Show resolved Hide resolved

lqiu96 commented Feb 27, 2024

View reviewed changes

chore: Use otel v1.34.1

4edded1

lqiu96 commented Feb 27, 2024

View reviewed changes

showcase/gapic-showcase/src/test/java/com/google/showcase/v1beta1/it/ITOtelMetrics.java Outdated Show resolved Hide resolved

chore: Ignore the HttpJson Otel tests

78cc64a

blakeli0 reviewed Mar 5, 2024

View reviewed changes

showcase/pom.xml Outdated Show resolved Hide resolved

lqiu96 added 2 commits March 5, 2024 11:45

chore: Address PR comments

6df9640

chore: Throw exception if operation completion call has been invoked …

edd7cea

…1+ times

lqiu96 commented Mar 5, 2024

View reviewed changes

gax-java/gax/src/main/java/com/google/api/gax/rpc/ClientSettings.java Outdated Show resolved Hide resolved

lqiu96 commented Mar 5, 2024

View reviewed changes

gax-java/gax/src/main/java/com/google/api/gax/rpc/ClientSettings.java Outdated Show resolved Hide resolved

lqiu96 added 4 commits March 5, 2024 16:35

chore: Address PR comments

32ee2fe

chore: Add otel bom to gapic-generator-java bom

825175e

chore: Address PR comments

09ea967

chore: Fix showcase tests

ea5a776

lqiu96 requested a review from blakeli0 March 5, 2024 23:08

blakeli0 reviewed Mar 6, 2024

View reviewed changes

gax-java/dependencies.properties Outdated Show resolved Hide resolved

blakeli0 reviewed Mar 6, 2024

View reviewed changes

lqiu96 and others added 3 commits March 6, 2024 10:41

Update gax-java/dependencies.properties

998eb68

Co-authored-by: Blake Li <[email protected]>

Merge branch 'main' into implement-OpentelemetryMetricsRecorder

1453461

chore: Use gax-java as instrument scope name

6c7bca3

blakeli0 reviewed Mar 13, 2024

View reviewed changes

gax-java/gax/src/main/java/com/google/api/gax/tracing/OpenTelemetryMetricsRecorder.java Show resolved Hide resolved

blakeli0 approved these changes Mar 13, 2024

View reviewed changes

blakeli0 added 2 commits March 14, 2024 15:14

Update gax-java/gax/src/main/java/com/google/api/gax/tracing/OpenTele…

fad8094

…metryMetricsRecorder.java

Merge branch 'main' into implement-OpentelemetryMetricsRecorder

41bbf57

blakeli0 added the automerge Merge the pull request once unit tests and other checks pass. label Mar 14, 2024

blakeli0 reviewed Mar 14, 2024

View reviewed changes

gax-java/gax/src/main/java/com/google/api/gax/tracing/OpenTelemetryMetricsRecorder.java Show resolved Hide resolved

Update gax-java/gax/src/main/java/com/google/api/gax/tracing/OpenTele…

a79e4e7

…metryMetricsRecorder.java

blakeli0 merged commit b936580 into main Mar 14, 2024
43 of 44 checks passed

blakeli0 deleted the implement-OpentelemetryMetricsRecorder branch March 14, 2024 20:17

gcf-merge-on-green bot removed the automerge Merge the pull request once unit tests and other checks pass. label Mar 14, 2024

release-please bot mentioned this pull request Mar 14, 2024

chore(main): release 2.38.0 #2529

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Introduce OpenTelemetry Metrics Recording #2500

feat: Introduce OpenTelemetry Metrics Recording #2500

lqiu96 commented Feb 23, 2024 •

edited by blakeli0

Loading

lqiu96 Feb 27, 2024

blakeli0 Mar 5, 2024 •

edited

Loading

blakeli0 Mar 5, 2024 •

edited

Loading

blakeli0 Mar 6, 2024

lqiu96 Mar 6, 2024

sonarqubecloud bot commented Mar 14, 2024

sonarqubecloud bot commented Mar 14, 2024

feat: Introduce OpenTelemetry Metrics Recording #2500

feat: Introduce OpenTelemetry Metrics Recording #2500

Conversation

lqiu96 commented Feb 23, 2024 • edited by blakeli0 Loading

lqiu96 Feb 27, 2024

Choose a reason for hiding this comment

blakeli0 Mar 5, 2024 • edited Loading

Choose a reason for hiding this comment

blakeli0 Mar 5, 2024 • edited Loading

Choose a reason for hiding this comment

blakeli0 Mar 6, 2024

Choose a reason for hiding this comment

lqiu96 Mar 6, 2024

Choose a reason for hiding this comment

sonarqubecloud bot commented Mar 14, 2024

Quality Gate failed for 'gapic-generator-java-root'

sonarqubecloud bot commented Mar 14, 2024

Quality Gate passed for 'java_showcase_integration_tests'

lqiu96 commented Feb 23, 2024 •

edited by blakeli0

Loading

blakeli0 Mar 5, 2024 •

edited

Loading

blakeli0 Mar 5, 2024 •

edited

Loading