Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add experimental S2A integration in client libraries grpc transport #3326

Merged
merged 59 commits into from
Nov 14, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
59 commits
Select commit Hold shift + click to select a range
aebd139
bump grpc versions.
rmehta19 Oct 29, 2024
b456935
bump auth to local SNAPSHOT.
rmehta19 Oct 29, 2024
3510643
implementation.
rmehta19 Nov 1, 2024
e799526
tests.
rmehta19 Nov 1, 2024
70ae8d0
use endpoint override.
rmehta19 Nov 1, 2024
7ef7313
fix typo
rmehta19 Nov 1, 2024
6d98115
cleanup.
rmehta19 Nov 4, 2024
250fecc
Merge branch 'main' into grpc-channel-using-s2a
rmehta19 Nov 4, 2024
8469a1a
resolve merge conflict with PR#1560
rmehta19 Nov 4, 2024
e880fbe
remove unnecessary dependency.
rmehta19 Nov 4, 2024
28adb31
add mtlsEndpoint to ClientSettings.
rmehta19 Nov 4, 2024
b3e2e06
add logs.
rmehta19 Nov 4, 2024
90892ef
remove ignore rule.
rmehta19 Nov 4, 2024
33b710c
separate into single condition if statements.
rmehta19 Nov 4, 2024
b320cc2
add logic to resolve endpointOverride / customEndpoint in EndpointCon…
rmehta19 Nov 4, 2024
035e17e
add javadocs.
rmehta19 Nov 4, 2024
759c3df
mtlsendpoint changes.
rmehta19 Nov 5, 2024
c096cb7
isS2AEnabled.
rmehta19 Nov 5, 2024
dc4b61e
expose endpointContext.
rmehta19 Nov 5, 2024
d4fcc71
Merge branch 'main' into grpc-channel-using-s2a
rmehta19 Nov 5, 2024
393190a
remove needsEndpointContext.
rmehta19 Nov 6, 2024
30a37c2
reduce nesting.
rmehta19 Nov 6, 2024
c3b93a0
check Strings null or empty.
rmehta19 Nov 6, 2024
18a4cf2
remove unecessary check.
rmehta19 Nov 6, 2024
0897730
set endpointContext in grpc transport provider tests.
rmehta19 Nov 6, 2024
9901656
package private.
rmehta19 Nov 6, 2024
257c515
set System::getenv as envProvider.
rmehta19 Nov 6, 2024
e4565f4
add javadoc.
rmehta19 Nov 6, 2024
33bd7a6
no getters for endpointcontext.
rmehta19 Nov 6, 2024
f9eef5b
nits + package private.
rmehta19 Nov 6, 2024
a7af12e
update javadocs + add mtlsServiceAddress.
rmehta19 Nov 6, 2024
eb70225
add javadocs.
rmehta19 Nov 6, 2024
2a50985
update readme.
rmehta19 Nov 6, 2024
943683d
private + readme updates.
rmehta19 Nov 6, 2024
34f030f
Merge branch 'main' into grpc-channel-using-s2a
rmehta19 Nov 6, 2024
fb3d608
update CLIRR-ignored-differences.
rmehta19 Nov 6, 2024
926faca
mark endpoint as Obsolete.
rmehta19 Nov 6, 2024
4ee2ee9
fix remaining tests to mock endpointContext.resolvedEndpoint().
rmehta19 Nov 6, 2024
2f70bf8
modify obsolete statement.
rmehta19 Nov 6, 2024
cb5768f
Merge branch 'main' into grpc-channel-using-s2a
rmehta19 Nov 7, 2024
c96c760
revert usage of endpointContext.resolvedEndpoint().
rmehta19 Nov 7, 2024
30ff3d6
Merge branch 'main' into grpc-channel-using-s2a
rmehta19 Nov 8, 2024
791ab2c
don't propogate endpointContext.
rmehta19 Nov 11, 2024
a4ee6c5
log as warning.
rmehta19 Nov 11, 2024
6eeccb2
package private + comment on S2A env var.
rmehta19 Nov 11, 2024
03eb991
package private useS2A.
rmehta19 Nov 12, 2024
91175ef
check if files exist.
rmehta19 Nov 12, 2024
04d47a0
default unsupported exception.
rmehta19 Nov 12, 2024
7d7b233
log a warning.
rmehta19 Nov 12, 2024
a42dcbd
pass File to TlsChannelCredentials.
rmehta19 Nov 12, 2024
56551c6
test createPlaintextS2AChannelCredentials.
rmehta19 Nov 12, 2024
1ff7a92
fix file checking logic.
rmehta19 Nov 12, 2024
2958fb4
document mtls mds.
rmehta19 Nov 13, 2024
c9a7edd
rename mtls mds vars to be clear these are filepaths.
rmehta19 Nov 13, 2024
8bf770d
bump auth dep and add some tests for createS2ASecuredChannelCredentials.
rmehta19 Nov 13, 2024
33faba1
Merge branch 'main' into grpc-channel-using-s2a
rmehta19 Nov 13, 2024
924a4e4
fix BUILD.
rmehta19 Nov 13, 2024
b00fd53
Merge branch 'main' into grpc-channel-using-s2a
rmehta19 Nov 14, 2024
ca5be85
fix: Do not build s2a frpm grpc's Bazel target.
blakeli0 Nov 14, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion WORKSPACE
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,6 @@ maven_install(
"com.google.api:gapic-generator-java:" + _gapic_generator_java_version,
] + PROTOBUF_MAVEN_ARTIFACTS + IO_GRPC_GRPC_JAVA_ARTIFACTS,
fail_on_missing_checksum = False,
override_targets = IO_GRPC_GRPC_JAVA_OVERRIDE_TARGETS,
repositories = [
"m2Local",
"https://repo.maven.apache.org/maven2/",
Expand Down
2 changes: 1 addition & 1 deletion gax-java/dependencies.properties
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ version.io_grpc=1.68.1
# 2) Replace all characters which are neither alphabetic nor digits with the underscore ('_') character
maven.com_google_api_grpc_proto_google_common_protos=com.google.api.grpc:proto-google-common-protos:2.46.0
maven.com_google_api_grpc_grpc_google_common_protos=com.google.api.grpc:grpc-google-common-protos:2.46.0
maven.com_google_auth_google_auth_library_oauth2_http=com.google.auth:google-auth-library-oauth2-http:1.29.0
maven.com_google_auth_google_auth_library_oauth2_http=com.google.auth:google-auth-library-oauth2-http:1.30.0
maven.com_google_auth_google_auth_library_credentials=com.google.auth:google-auth-library-credentials:1.30.0
maven.io_opentelemetry_opentelemetry_api=io.opentelemetry:opentelemetry-api:1.42.1
maven.io_opencensus_opencensus_api=io.opencensus:opencensus-api:0.31.1
Expand Down
1 change: 1 addition & 0 deletions gax-java/gax-grpc/BUILD.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ _COMPILE_DEPS = [
"@io_grpc_grpc_netty_shaded//jar",
"@io_grpc_grpc_grpclb//jar",
"@io_grpc_grpc_java//alts:alts",
"@io_grpc_grpc_java//s2a:s2av2_credentials",
"@io_netty_netty_tcnative_boringssl_static//jar",
"@javax_annotation_javax_annotation_api//jar",
"//gax:gax",
Expand Down
4 changes: 4 additions & 0 deletions gax-java/gax-grpc/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,10 @@
<groupId>io.grpc</groupId>
<artifactId>grpc-protobuf</artifactId>
</dependency>
<dependency>
<groupId>io.grpc</groupId>
<artifactId>grpc-s2a</artifactId>
</dependency>
rmehta19 marked this conversation as resolved.
Show resolved Hide resolved
<dependency>
<groupId>io.grpc</groupId>
<artifactId>grpc-stub</artifactId>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -46,19 +46,24 @@
import com.google.auth.ApiKeyCredentials;
import com.google.auth.Credentials;
import com.google.auth.oauth2.ComputeEngineCredentials;
import com.google.auth.oauth2.SecureSessionAgent;
import com.google.auth.oauth2.SecureSessionAgentConfig;
import com.google.common.annotations.VisibleForTesting;
import com.google.common.base.Preconditions;
import com.google.common.base.Strings;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.ImmutableMap;
import com.google.common.io.Files;
import io.grpc.CallCredentials;
import io.grpc.ChannelCredentials;
import io.grpc.Grpc;
import io.grpc.InsecureChannelCredentials;
import io.grpc.ManagedChannel;
import io.grpc.ManagedChannelBuilder;
import io.grpc.TlsChannelCredentials;
import io.grpc.alts.GoogleDefaultChannelCredentials;
import io.grpc.auth.MoreCallCredentials;
import io.grpc.s2a.S2AChannelCredentials;
import java.io.File;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
Expand Down Expand Up @@ -99,6 +104,15 @@ public final class InstantiatingGrpcChannelProvider implements TransportChannelP
@VisibleForTesting
static final String DIRECT_PATH_ENV_ENABLE_XDS = "GOOGLE_CLOUD_ENABLE_DIRECT_PATH_XDS";

// The public portion of the mTLS MDS root certificate is stored for performing
// cert verification when establishing an mTLS connection with the MDS. See
// https://cloud.google.com/compute/docs/metadata/overview#https-mds-root-certs
private static final String MTLS_MDS_ROOT_PATH = "/run/google-mds-mtls/root.crt";
// The mTLS MDS credentials are formatted as the concatenation of a PEM-encoded certificate chain
// followed by a PEM-encoded private key. See
// https://cloud.google.com/compute/docs/metadata/overview#https-mds-client-certs
private static final String MTLS_MDS_CERT_CHAIN_AND_KEY_PATH = "/run/google-mds-mtls/client.key";

static final long DIRECT_PATH_KEEP_ALIVE_TIME_SECONDS = 3600;
static final long DIRECT_PATH_KEEP_ALIVE_TIMEOUT_SECONDS = 20;
static final String GCE_PRODUCTION_NAME_PRIOR_2016 = "Google";
Expand All @@ -107,6 +121,7 @@ public final class InstantiatingGrpcChannelProvider implements TransportChannelP
private final int processorCount;
private final Executor executor;
private final HeaderProvider headerProvider;
private final boolean useS2A;
private final String endpoint;
// TODO: remove. envProvider currently provides DirectPath environment variable, and is only used
// during initial rollout for DirectPath. This provider will be removed once the DirectPath
Expand All @@ -126,6 +141,7 @@ public final class InstantiatingGrpcChannelProvider implements TransportChannelP
@Nullable private final Boolean allowNonDefaultServiceAccount;
@VisibleForTesting final ImmutableMap<String, ?> directPathServiceConfig;
@Nullable private final MtlsProvider mtlsProvider;
@Nullable private final SecureSessionAgent s2aConfigProvider;
@VisibleForTesting final Map<String, String> headersWithDuplicatesRemoved = new HashMap<>();

@Nullable
Expand All @@ -136,7 +152,9 @@ private InstantiatingGrpcChannelProvider(Builder builder) {
this.executor = builder.executor;
this.headerProvider = builder.headerProvider;
this.endpoint = builder.endpoint;
this.useS2A = builder.useS2A;
this.mtlsProvider = builder.mtlsProvider;
this.s2aConfigProvider = builder.s2aConfigProvider;
this.envProvider = builder.envProvider;
this.interceptorProvider = builder.interceptorProvider;
this.maxInboundMessageSize = builder.maxInboundMessageSize;
Expand Down Expand Up @@ -225,6 +243,17 @@ public TransportChannelProvider withEndpoint(String endpoint) {
return toBuilder().setEndpoint(endpoint).build();
}

/**
* Specify whether or not to use S2A.
*
* @param useS2A
* @return A new {@link InstantiatingGrpcChannelProvider} with useS2A set.
*/
@Override
public TransportChannelProvider withUseS2A(boolean useS2A) {
return toBuilder().setUseS2A(useS2A).build();
}

/** @deprecated Please modify pool settings via {@link #toBuilder()} */
@Deprecated
@Override
Expand Down Expand Up @@ -410,6 +439,101 @@ ChannelCredentials createMtlsChannelCredentials() throws IOException, GeneralSec
return null;
}

/**
* This method creates {@link TlsChannelCredentials} to be used by the client to establish an mTLS
* connection to S2A. Returns null if any of {@param trustBundle}, {@param privateKey} or {@param
* certChain} are missing.
*
* @param trustBundle the trust bundle to be used to establish the client -> S2A mTLS connection
* @param privateKey the client's private key to be used to establish the client -> S2A mtls
* connection
* @param certChain the client's cert chain to be used to establish the client -> S2A mtls
* connection
* @return {@link ChannelCredentials} to use to create an mtls connection between client and S2A
* @throws IOException on error
*/
@VisibleForTesting
ChannelCredentials createMtlsToS2AChannelCredentials(
File trustBundle, File privateKey, File certChain) throws IOException {
if (trustBundle == null || privateKey == null || certChain == null) {
return null;
}
return TlsChannelCredentials.newBuilder()
.keyManager(privateKey, certChain)
blakeli0 marked this conversation as resolved.
Show resolved Hide resolved
.trustManager(trustBundle)
.build();
}

/**
* This method creates {@link ChannelCredentials} to be used by client to establish a plaintext
* connection to S2A. if {@param plaintextAddress} is not present, returns null.
*
* @param plaintextAddress the address to reach S2A which accepts plaintext connections
* @return {@link ChannelCredentials} to use to create a plaintext connection between client and
* S2A
*/
ChannelCredentials createPlaintextToS2AChannelCredentials(String plaintextAddress) {
if (Strings.isNullOrEmpty(plaintextAddress)) {
return null;
}
return S2AChannelCredentials.newBuilder(plaintextAddress, InsecureChannelCredentials.create())
.build();
}

/**
* This method creates gRPC {@link ChannelCredentials} configured to use S2A to estbalish a mTLS
* connection. First, the address of S2A is discovered by using the {@link S2A} utility to learn
* the {@code mtlsAddress} to reach S2A and the {@code plaintextAddress} to reach S2A. Prefer to
* use the {@code mtlsAddress} address to reach S2A if it is non-empty and the MTLS-MDS
* credentials can successfully be discovered and used to create {@link TlsChannelCredentials}. If
* there is any failure using mTLS-to-S2A, fallback to using a plaintext connection to S2A using
* the {@code plaintextAddress}. If {@code plaintextAddress} is not available, this function
* returns null; in this case S2A will not be used, and a TLS connection to the service will be
* established.
*
* @return {@link ChannelCredentials} configured to use S2A to create mTLS connection to
* mtlsEndpoint.
*/
ChannelCredentials createS2ASecuredChannelCredentials() {
lqiu96 marked this conversation as resolved.
Show resolved Hide resolved
SecureSessionAgentConfig config = s2aConfigProvider.getConfig();
String plaintextAddress = config.getPlaintextAddress();
String mtlsAddress = config.getMtlsAddress();
if (Strings.isNullOrEmpty(mtlsAddress)) {
// Fallback to plaintext connection to S2A.
LOG.log(
Level.INFO,
"Cannot establish an mTLS connection to S2A because autoconfig endpoint did not return a mtls address to reach S2A.");
return createPlaintextToS2AChannelCredentials(plaintextAddress);
}
// Currently, MTLS to MDS is only available on GCE. See:
// https://cloud.google.com/compute/docs/metadata/overview#https-mds
// Try to load MTLS-MDS creds.
File rootFile = new File(MTLS_MDS_ROOT_PATH);
File certKeyFile = new File(MTLS_MDS_CERT_CHAIN_AND_KEY_PATH);
if (rootFile.isFile() && certKeyFile.isFile()) {
// Try to connect to S2A using mTLS.
ChannelCredentials mtlsToS2AChannelCredentials = null;
try {
mtlsToS2AChannelCredentials =
createMtlsToS2AChannelCredentials(rootFile, certKeyFile, certKeyFile);
} catch (IOException ignore) {
// Fallback to plaintext-to-S2A connection on error.
LOG.log(
Level.WARNING,
"Cannot establish an mTLS connection to S2A due to error creating MTLS to MDS TlsChannelCredentials credentials, falling back to plaintext connection to S2A: "
+ ignore.getMessage());
return createPlaintextToS2AChannelCredentials(plaintextAddress);
}
return S2AChannelCredentials.newBuilder(mtlsAddress, mtlsToS2AChannelCredentials).build();
} else {
// Fallback to plaintext-to-S2A connection if MTLS-MDS creds do not exist.
LOG.log(
Level.INFO,
blakeli0 marked this conversation as resolved.
Show resolved Hide resolved
"Cannot establish an mTLS connection to S2A because MTLS to MDS credentials do not exist on filesystem, falling back to plaintext connection to S2A");
return createPlaintextToS2AChannelCredentials(plaintextAddress);
}
Comment on lines +528 to +534
Copy link
Contributor

@lqiu96 lqiu96 Nov 14, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Perhaps this else block could be an early return instead
i.e.

if (!rootFile.isFile() || !certKeyFile.isFile()) {
...
}
{Logic to create s2a}

would reduce the nesting. Ignore this comment if this was already discussed.

PR LGTM

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, I also suggested it in #3326 (comment), but it does not have to block this PR.

}

private ManagedChannel createSingleChannel() throws IOException {
GrpcHeaderInterceptor headerInterceptor =
new GrpcHeaderInterceptor(headersWithDuplicatesRemoved);
Expand Down Expand Up @@ -447,16 +571,31 @@ private ManagedChannel createSingleChannel() throws IOException {
builder.keepAliveTime(DIRECT_PATH_KEEP_ALIVE_TIME_SECONDS, TimeUnit.SECONDS);
builder.keepAliveTimeout(DIRECT_PATH_KEEP_ALIVE_TIMEOUT_SECONDS, TimeUnit.SECONDS);
} else {
// Try and create credentials via DCA. See https://google.aip.dev/auth/4114.
ChannelCredentials channelCredentials;
try {
channelCredentials = createMtlsChannelCredentials();
} catch (GeneralSecurityException e) {
throw new IOException(e);
}
if (channelCredentials != null) {
// Create the channel using channel credentials created via DCA.
builder = Grpc.newChannelBuilder(endpoint, channelCredentials);
} else {
builder = ManagedChannelBuilder.forAddress(serviceAddress, port);
// Could not create channel credentials via DCA. In accordance with
// https://google.aip.dev/auth/4115, if credentials not available through
// DCA, try mTLS with credentials held by the S2A (Secure Session Agent).
if (useS2A) {
channelCredentials = createS2ASecuredChannelCredentials();
}
if (channelCredentials != null) {
// Create the channel using S2A-secured channel credentials.
// {@code endpoint} is set to mtlsEndpoint in {@link EndpointContext} when useS2A is true.
builder = Grpc.newChannelBuilder(endpoint, channelCredentials);
} else {
// Use default if we cannot initialize channel credentials via DCA or S2A.
builder = ManagedChannelBuilder.forAddress(serviceAddress, port);
}
}
}
// google-c2p resolver requires service config lookup
Expand Down Expand Up @@ -604,7 +743,9 @@ public static final class Builder {
private Executor executor;
private HeaderProvider headerProvider;
private String endpoint;
private boolean useS2A;
private EnvironmentProvider envProvider;
private SecureSessionAgent s2aConfigProvider = SecureSessionAgent.create();
private MtlsProvider mtlsProvider = new MtlsProvider();
@Nullable private GrpcInterceptorProvider interceptorProvider;
@Nullable private Integer maxInboundMessageSize;
Expand Down Expand Up @@ -632,6 +773,7 @@ private Builder(InstantiatingGrpcChannelProvider provider) {
this.executor = provider.executor;
this.headerProvider = provider.headerProvider;
this.endpoint = provider.endpoint;
this.useS2A = provider.useS2A;
this.envProvider = provider.envProvider;
this.interceptorProvider = provider.interceptorProvider;
this.maxInboundMessageSize = provider.maxInboundMessageSize;
Expand All @@ -648,6 +790,7 @@ private Builder(InstantiatingGrpcChannelProvider provider) {
this.allowNonDefaultServiceAccount = provider.allowNonDefaultServiceAccount;
this.directPathServiceConfig = provider.directPathServiceConfig;
this.mtlsProvider = provider.mtlsProvider;
this.s2aConfigProvider = provider.s2aConfigProvider;
}

/**
Expand Down Expand Up @@ -700,12 +843,23 @@ public Builder setEndpoint(String endpoint) {
return this;
}

Builder setUseS2A(boolean useS2A) {
this.useS2A = useS2A;
return this;
}

@VisibleForTesting
Builder setMtlsProvider(MtlsProvider mtlsProvider) {
this.mtlsProvider = mtlsProvider;
return this;
}

@VisibleForTesting
Builder setS2AConfigProvider(SecureSessionAgent s2aConfigProvider) {
this.s2aConfigProvider = s2aConfigProvider;
return this;
}

/**
* Sets the GrpcInterceptorProvider for this TransportChannelProvider.
*
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -101,6 +101,8 @@ void setUp() throws IOException {
TransportChannel transportChannel =
GrpcTransportChannel.newBuilder().setManagedChannel(channel).build();
when(operationsChannelProvider.getTransportChannel()).thenReturn(transportChannel);
when(operationsChannelProvider.withUseS2A(Mockito.any(boolean.class)))
.thenReturn(operationsChannelProvider);

clock = new FakeApiClock(0L);
executor = RecordingScheduler.create(clock);
Expand Down
Loading
Loading