Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: [Spanner] Add query options support #4512

Merged
merged 29 commits into from
Mar 14, 2020
Merged
Show file tree
Hide file tree
Changes from 15 commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
33c056a
Added QueryOptions class and tests.
skuruppu Feb 25, 2020
00fd37f
Added QueryOptions property to a SpannerConnection.
skuruppu Feb 25, 2020
71c7447
Added method to get proto of QueryOptions.
skuruppu Feb 26, 2020
38d1754
Added QueryOptions property to SpannerCommand.
skuruppu Feb 26, 2020
33259c0
Set QueryOptions when ExecutableCommand is created.
skuruppu Feb 26, 2020
7ee5116
Pass QueryOptions from the connection to SpannerCommand.
skuruppu Feb 26, 2020
b6262ea
Added equality checking for QueryOptions.
skuruppu Feb 26, 2020
f332fc7
Added unit tests for QueryOptions.
skuruppu Feb 26, 2020
3463c91
Fixes to QueryOptions based on review feedback.
skuruppu Feb 28, 2020
3d35057
Fixes QueryOptions calls based on immutability constraint.
skuruppu Mar 5, 2020
68293b2
Implemented GetEffectiveQueryOptions().
skuruppu Mar 6, 2020
d5cea1b
Simplified QueryOptions setting in SpannerCommand.
skuruppu Mar 6, 2020
6184037
Simplified QueryOptions setting in SpannerConnection.
skuruppu Mar 6, 2020
0ebebb3
Removed ToProto() since it's no longer used.
skuruppu Mar 6, 2020
5cd9d38
Started fixing tests to check QueryOptions in the client.
skuruppu Mar 6, 2020
299ffa6
Fixed the unit tests to use mocks.
skuruppu Mar 11, 2020
4da2633
Revert "Removed ToProto() since it's no longer used."
skuruppu Mar 11, 2020
cf6b811
Simplified GetEffectiveQueryOptions().
skuruppu Mar 11, 2020
c1f2cd5
Fixes according to review feedback.
skuruppu Mar 11, 2020
032f668
Removed unnecessary line-wrapping.
skuruppu Mar 11, 2020
becf5e4
Updated the env var name to be more descriptive.
skuruppu Mar 11, 2020
9a3f2df
Revert "Updated the env var name to be more descriptive."
skuruppu Mar 11, 2020
ff642e9
Fixes according to review feedback.
skuruppu Mar 12, 2020
06bb864
Added integration tests for QueryOptions.
skuruppu Mar 12, 2020
94cb5ad
Added an IT for query-level options setting.
skuruppu Mar 13, 2020
c8f2de2
Refactored SpannerCommand tests based on feedback.
skuruppu Mar 13, 2020
15221a5
Added an Empty initializer to QueryOptions.
skuruppu Mar 13, 2020
1b59e1b
Clone QueryOptions when cloning SpannerCommand.
skuruppu Mar 13, 2020
ec4b8c5
Fixed formatting issue.
skuruppu Mar 13, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
// Copyright 2020 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

using Google.Cloud.Spanner.V1;
using System;
using Xunit;

namespace Google.Cloud.Spanner.Data.Tests
{
public class QueryOptionsTests
{
[Fact]
public void FromProtoFromEmptyProto()
{
var proto = new V1.ExecuteSqlRequest.Types.QueryOptions();
var queryOptions = QueryOptions.FromProto(proto);
Assert.Equal("", queryOptions.OptimizerVersion);
}

[Fact]
public void NoOptionsSet()
{
var queryOptions = new QueryOptions();
Assert.Equal("", queryOptions.OptimizerVersion);
}

[Fact]
public void SetAndGetOptimizerVersion()
{
var queryOptions = new QueryOptions().WithOptimizerVersion("latest");
Assert.Equal("latest", queryOptions.OptimizerVersion);
}
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,15 @@
// See the License for the specific language governing permissions and
// limitations under the License.

using Google.Api.Gax.Grpc;
using Google.Cloud.Spanner.V1;
using Google.Cloud.Spanner.V1.Tests;
using Google.Cloud.Spanner.V1.Internal.Logging;
using Moq;
using System;
using System.Data;
using System.Linq;
using System.Threading.Tasks;
using Xunit;

namespace Google.Cloud.Spanner.Data.Tests
Expand Down Expand Up @@ -85,5 +91,100 @@ public void CloneUpdateWithParameters()
Assert.True(command.Parameters.SequenceEqual(command2.Parameters));
}

/*
skuruppu marked this conversation as resolved.
Show resolved Hide resolved
[Fact]
public void CommandHasConnectionQueryOptions()
{
const string optimzerVersion = "1";
var spannerClientMock = SpannerClientHelpers
.CreateMockClient(Logger.DefaultLogger, MockBehavior.Strict)
.Setup(client => client.ExecuteSqlAsync(It.IsNotNull<ExecuteSqlRequest>(), It.Is<ExecuteSqlRequest>(request => request.QueryOptions.OptimzerVersion == optimizerVersion)))
.Returns<ExecuteSqlRequest, CallSettings>((request, _) =>
{
ResultSet response = new ResultSet();

return Task.FromResult(response);
});

SpannerConnection connection = BuildSpannerConnection(spannerClientMock);
var queryOptions = new QueryOptions().WithOptimizerVersion(optimizerVersion);
connection.QueryOptions = queryOptions;

var command = connection.CreateSelectCommand("SELECT * FROM FOO");
using (var reader = await command.ExecuteReaderAsync())
{
while (await reader.ReadAsync())
{
// Do nothing.
}
}
}

[Fact]
public void CommandHasOptimizerVersionFromEnvironment()
{
// Save existing value of environment variable.
const string optimizerVersionVariable = "SPANNER_OPTIMIZER_VERSION";
string savedOptimizerVersion = Environment.GetEnvironmentVariable(optimizerVersionVariable);
const string envOptimizerVersion = "2";
Environment.SetEnvironmentVariable(optimizerVersionVariable, envOptimizerVersion);

var connection = new SpannerConnection("Data Source=projects/p/instances/i/databases/d");
connection.QueryOptions = new QueryOptions().WithOptimizerVersion("1");

var command = connection.CreateSelectCommand("SELECT * FROM FOO");
// Optimizer version set through environment variable has higher
// precedence than version set through connection.
Assert.Equal(envOptimizerVersion, command.QueryOptions.OptimizerVersion);

// Set the environment back.
Environment.SetEnvironmentVariable(optimizerVersionVariable, savedOptimizerVersion);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be good to do this in a finally block. You might even want to write a method accepting an Action that will set the environment variable, execute the action, then reset the environment variable.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool, didn't know about Action. Updated to how I think actions should be used. Also using a finally block now.

}

[Fact]
public void CommandHasOptimizerVersionSetOnCommand()
{
// Save existing value of environment variable.
const string optimizerVersionVariable = "SPANNER_OPTIMIZER_VERSION";
string savedOptimizerVersion = Environment.GetEnvironmentVariable(optimizerVersionVariable);
const string envOptimizerVersion = "2";
Environment.SetEnvironmentVariable(optimizerVersionVariable, envOptimizerVersion);

var connection = new SpannerConnection("Data Source=projects/p/instances/i/databases/d");
connection.QueryOptions = new QueryOptions().WithOptimizerVersion("1");

var command = connection.CreateSelectCommand("SELECT * FROM FOO");
var commandOptimizerVersion = "3";
command.QueryOptions = new QueryOptions().WithOptimizerVersion("3");
// Optimizer version set at a command level has higher precedence
// than version set through the connection or the environment
// variable.
Assert.Equal(commandOptimizerVersion, command.QueryOptions.OptimizerVersion);

// Set the environment back.
Environment.SetEnvironmentVariable(optimizerVersionVariable, savedOptimizerVersion);
}

private SpannerConnection BuildSpannerConnection(Mock<SpannerClient> spannerClientMock)
{
var spannerClient = spannerClientMock.Object;
var sessionPoolOptions = new SessionPoolOptions
{
MaintenanceLoopDelay = TimeSpan.Zero
};

var sessionPoolManager = new SessionPoolManager(sessionPoolOptions, spannerClient.Settings.Logger, (_o, _s, _l) => Task.FromResult(spannerClient));
sessionPoolManager.SpannerSettings.Scheduler = spannerClient.Settings.Scheduler;
sessionPoolManager.SpannerSettings.Clock = spannerClient.Settings.Clock;

SpannerConnectionStringBuilder builder = new SpannerConnectionStringBuilder
{
DataSource = DatabaseName.Format(SpannerClientHelpers.ProjectId, SpannerClientHelpers.Instance, SpannerClientHelpers.Database),
SessionPoolManager = sessionPoolManager
};

return new SpannerConnection(builder);
}
*/
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
// Copyright 2020 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

using Google.Api.Gax;
using Google.Cloud.Spanner.V1;
using System;

namespace Google.Cloud.Spanner.Data
{
/// <summary>
/// Immutable class representing query options.
/// </summary>
public sealed class QueryOptions : IEquatable<QueryOptions>
{
/// <summary>
/// The query optimizer version configured in the options.
/// </summary>
public string OptimizerVersion
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: simplify to

public string OptimizerVersion => Proto.OptimizerVersion;

{
get => Proto.OptimizerVersion;
}

/// <summary>
/// Clones the options and sets the optimizer version to the given value.
/// </summary>
/// <returns>
/// A clone of the options with the updated optimizer version.
/// </returns>
/// <remarks>
/// <para>The parameter allows individual queries to pick different query
/// optimizer versions.</para>
/// <para>Specifying "latest" as a value instructs Cloud Spanner to use the
/// latest supported query optimizer version. If not specified, Cloud Spanner
/// uses optimizer version set at the database level options. Any other
/// positive integer (from the list of supported optimizer versions)
/// overrides the default optimizer version for query execution.</para>
/// </remarks>
/// <param name="optimizerVersion">Optimizer version to set.</param>
public QueryOptions WithOptimizerVersion(string optimizerVersion)
{
var protoCopy = Proto.Clone();
protoCopy.OptimizerVersion = optimizerVersion;
return new QueryOptions(protoCopy);
}

skuruppu marked this conversation as resolved.
Show resolved Hide resolved

/// <summary>
/// The proto representation of the query options. Must not be mutated
/// or exposed publicly.
/// </summary>
internal V1.ExecuteSqlRequest.Types.QueryOptions Proto { get; }

private QueryOptions(V1.ExecuteSqlRequest.Types.QueryOptions proto) => Proto = proto;

/// <summary>
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An optional alternative to this would be to have a static Empty property, e.g.

public static QueryOptions Empty { get; } = new QueryOptions(new V1.ExecuteSqlRequest.Types.QueryOptions());

Then code which is currently new QueryOptions().WithOptimizerVersion(...) would become QueryOptions.Empty.WithOptimizerVersion(...) which might be clearer/simpler.

+amanda-tarafa do you have a preference on this one?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like this idea because I find it to be more readable so I changed it.

/// Creates query options without specifying any options.
/// </summary>
public QueryOptions() : this(new V1.ExecuteSqlRequest.Types.QueryOptions())
{
}

/// <summary>
/// Set query options from the given proto.
/// </summary>
/// <remarks>
/// The given proto should not be null. The given proto is cloned.
/// </remarks>
/// <param name="proto">The proto to construct <see cref="QueryOptions"/> from.</param>
public static QueryOptions FromProto(
V1.ExecuteSqlRequest.Types.QueryOptions proto)
{
GaxPreconditions.CheckNotNull(proto, nameof(proto));
return new QueryOptions(proto.Clone());
}

/// <inheritdoc />
public bool Equals(QueryOptions other)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd probably write this as:

public bool Equals(QueryOptions other) => other is object && OptimizerVersion == other.OptimizerVersion;

(As other.OptimizerVersion can never return null, and OptimizerVersion can never be null, you could just return OptimizerVersion == other?.OptimizerVersion, but the reasoning required makes that harder to understand.)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's neat but I agree that's harder to reason about. But I made the first change you suggested.

{
if (other is null)
{
return false;
}
return OptimizerVersion == other.OptimizerVersion;
}

/// <inheritdoc />
public override bool Equals(object obj) => Equals(obj as QueryOptions);

/// <inheritdoc />
public override int GetHashCode() => Proto.GetHashCode();
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -53,12 +53,15 @@ private static DatabaseAdminSettings CreateDatabaseAdminSettings()
return settings;
}

private const string SpannerOptimizerVersionVariable = "SPANNER_OPTIMIZER_VERSION";

internal SpannerConnection Connection { get; }
internal SpannerCommandTextBuilder CommandTextBuilder { get; }
internal int CommandTimeout { get; }
internal SpannerTransaction Transaction { get; }
internal CommandPartition Partition { get; }
internal SpannerParameterCollection Parameters { get; }
internal QueryOptions QueryOptions { get; }

public ExecutableCommand(SpannerCommand command)
{
Expand All @@ -68,6 +71,7 @@ public ExecutableCommand(SpannerCommand command)
Partition = command.Partition;
Parameters = command.Parameters;
Transaction = command._transaction;
QueryOptions = command.QueryOptions;
}

// ExecuteScalar is simply implemented in terms of ExecuteReader.
Expand Down Expand Up @@ -335,6 +339,43 @@ private List<Mutation> GetMutations()
}
}

// Based on the QueryOptions set at various levels (connection, environment and command),
// constructs the QueryOptions proto to set in the ExecuteSqlRequest.
// Options set at the SpannerCommand-level has the highest precedence.
// Options set at the environment variable level has the next highest precedence.
// Options set at the connection level has the lowest precedence.
private V1.ExecuteSqlRequest.Types.QueryOptions GetEffectiveQueryOptions()
{
string optimizerVersion = "";

// Query options set at the command level have the highest precedence.
if (QueryOptions != null)
{
optimizerVersion = QueryOptions.OptimizerVersion;
}

// Query options set through an environment variable have the next highest precedence.
if (string.IsNullOrEmpty(optimizerVersion))
{
optimizerVersion = Environment.GetEnvironmentVariable(SpannerOptimizerVersionVariable)?.Trim() ?? "";
}

// Query options set through the connection have the lowest highest precedence.
if (string.IsNullOrEmpty(optimizerVersion) && Connection.QueryOptions != null)
{
optimizerVersion = Connection.QueryOptions.OptimizerVersion;
}

if (string.IsNullOrEmpty(optimizerVersion))
{
return null;
}

var queryOptionsProto = new V1.ExecuteSqlRequest.Types.QueryOptions();
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can be simplified to use an object initializer:

return new V1.ExecuteSqlRequest.Types.QueryOptions { OptimizerVersion = optimizerVersion };

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

I actually simplified the whole function and made it a bit more future-proof by using the proto merge functionality. That way when we add new options, the merging will happen automatically. The only thing to update manually would be reading the new env vars.

queryOptionsProto.OptimizerVersion = optimizerVersion;
return queryOptionsProto;
}

private ExecuteSqlRequest GetExecuteSqlRequest()
{
if (Partition != null)
Expand All @@ -347,6 +388,8 @@ private ExecuteSqlRequest GetExecuteSqlRequest()
Sql = CommandTextBuilder.ToString()
};

request.QueryOptions = GetEffectiveQueryOptions();
skuruppu marked this conversation as resolved.
Show resolved Hide resolved

// See comment at the start of GetMutations.
SpannerConversionOptions options = null;
Parameters.FillSpannerCommandParams(out var parameters, request.ParamTypes, options);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ public sealed partial class SpannerCommand : DbCommand, ICloneable
private readonly CancellationTokenSource _synchronousCancellationTokenSource = new CancellationTokenSource();
private int _commandTimeout;
private SpannerTransaction _transaction;
private QueryOptions _queryOptions = null;

/// <summary>
/// Initializes a new instance of <see cref="SpannerCommand"/>, using a default command timeout.
Expand Down Expand Up @@ -219,6 +220,15 @@ public override UpdateRowSource UpdatedRowSource
}
}

/// <summary>
/// Query options to use when running SQL and streaming SQL commands.
/// </summary>
public QueryOptions QueryOptions
skuruppu marked this conversation as resolved.
Show resolved Hide resolved
{
get => _queryOptions;
set => _queryOptions = (QueryOptions) value;
skuruppu marked this conversation as resolved.
Show resolved Hide resolved
}

/// <inheritdoc />
protected override DbConnection DbConnection
{
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,18 @@ public override ConnectionState State

internal bool IsOpen => (State & ConnectionState.Open) == ConnectionState.Open;

private QueryOptions _queryOptions = null;

/// <summary>
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently this is only available as a property in the connection. Do you think we want people to be able to provide it in the connection string itself? I don't know what the expected use is likely to be.

We could potentially do that later.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At the moment, I'm not sure. In contexts like the JDBC driver, it is specified in the connection string.

I would be happy to do this in a separate PR.

/// Query options to use throughout the lifetime of the connection when
/// running SQL and streaming SQL requests.
/// </summary>
public QueryOptions QueryOptions
{
get => _queryOptions;
set => _queryOptions = (QueryOptions) value;
skuruppu marked this conversation as resolved.
Show resolved Hide resolved
}

/// <summary>
/// Creates a SpannerConnection with no datasource or credential specified.
/// </summary>
Expand Down