External watch API #2034

aaronlehmann · 2017-03-14T00:18:24Z

Add the first RPC from the external store API: Watch.

This RPC supports resuming from a known location, as well as starting the watch stream to only receive future events.

It can optionally return the old version of the object as well as the new one for Updates.

stevvooe · 2017-03-14T00:26:21Z

api/store.proto

+		StoreActionKind action = 1;
+
+		// Matched object
+		Object object = 2;


Can these be new and old or does that collide with a keyword?

I think that would be overly specific to the "update" case. For example, if we're looking at a deletion, new wouldn't be the right field name for the object that was deleted.

codecov · 2017-03-14T00:36:26Z

Codecov Report

Merging #2034 into master will increase coverage by 0.07%.
The diff coverage is 73.8%.

@@            Coverage Diff             @@
##           master    #2034      +/-   ##
==========================================
+ Coverage   54.28%   54.36%   +0.07%     
==========================================
  Files         111      113       +2     
  Lines       19327    19448     +121     
==========================================
+ Hits        10492    10572      +80     
- Misses       7569     7601      +32     
- Partials     1266     1275       +9

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6c388de...9472ac4. Read the comment docs.

aaronlehmann · 2017-03-17T17:44:39Z

Rebased

dongluochen · 2017-03-17T17:56:03Z

Is it easy to make StoreActionKind a bitmap combination? For watch request, user can specify all events for service with StoreActionKindCreate | StoreActionKindUpdate | StoreActionKindRemove?

StoreActionKindCreate  StoreActionKind = 1
StoreActionKindUpdate  StoreActionKind = 2
StoreActionKindRemove  StoreActionKind = 4

aaronlehmann · 2017-03-17T19:21:38Z

Is it easy to make StoreActionKind a bitmap combination?

Good idea. Done in the most recent commit. Note that StoreActionKind can't be changed because it's part of the wire format, so this introduces a new WatchActionKind.

dongluochen · 2017-03-17T23:41:28Z

api/store.proto

+// Watch. Note that the first item of this stream will always be a WatchMessage
+// with a nil Object, to signal that the stream has started.
+message WatchMessage {
+	message Event {


Each Event should have a timestamp of commit time to store.

Hm, this isn't currently available for historic events. For current Create and Update events, the individual objects will have the timestamp in Meta (but Delete events have the old timestamp).

Let's use the Meta timestamps.

Should we omit it for delete events or include it even though it's the timestamp of the last update?

Each event needs a timestamp. I'll use the last UpdatedAt of delete events for now. Another option is to use the current time of machine.

aluzzardi · 2017-03-18T00:10:10Z

api/store.proto

@@ -0,0 +1,152 @@
+syntax = "proto3";


I'd add a comment to everything that's not used

Not sure I follow. This isn't used yet - it's being added so that events can use it.

aluzzardi · 2017-03-18T00:22:27Z

api/store.proto

+	// previous version of the object on updates. Note that only live
+	// changes will include the old object (not historical changes
+	// retrieved using ResumeFrom).
+	bool include_old_object = 3;


Why do we need for this to be an option rather than the default?

@dongluochen was concerned about wasted cycles and bandwidth sending two copies of the object for each change.

For Event, it need to include the old object to find the difference.

If an application just needs the current state and change as trigger, old object is not required. For example, if an application only cares about create and remove events. it doesn't need old object.

aaronlehmann · 2017-03-18T00:55:29Z

Rebased on #2039, and added code generation for all type-specific parts of this code. Now the only code in watch.go is a short gRPC stream endpoint. The actual code to build selectors and convert events is all generated.

aaronlehmann · 2017-03-21T17:36:50Z

Rebased

dongluochen · 2017-03-22T01:54:34Z

LGTM

dongluochen · 2017-03-24T17:08:17Z

Ping @docker/core-swarmkit-maintainers.

This is the first part of an external store API. Watch returns a stream of new, updated, or removed objects matching the given selectors. Signed-off-by: Aaron Lehmann <[email protected]>

This adds a Version to every WatchMessage, and a ResumeFrom field in the initial WatchRequest. A client can specify its last seen version to avoid doing a complete resync. Signed-off-by: Aaron Lehmann <[email protected]>

Signed-off-by: Aaron Lehmann <[email protected]>

aaronlehmann · 2017-03-24T17:52:27Z

Rebased

diogomonica · 2017-03-24T21:46:48Z

@aaronlehmann do we filter secret data before we send those objects out?

aaronlehmann · 2017-03-24T21:54:54Z

No, this is not doing any redaction. It's the first piece of a store API, that will let external users interact with the data store at a lower level than the control API. I expect in the future there may be use cases where external controllers create and interact with secrets. We could always add redaction now, but might have to reconsider it later.

aluzzardi · 2017-03-30T01:04:29Z

protobuf/plugin/plugin.proto

 message StoreObject {
+	required WatchSelectors watch_selectors = 1;


is required still a thing in proto3?

This file doesn't use proto3 for whatever reason.

aluzzardi · 2017-03-30T01:07:36Z

LGTM

aaronlehmann force-pushed the watch-api branch from 1f3f13c to b5738c0 Compare March 14, 2017 00:25

stevvooe reviewed Mar 14, 2017

View reviewed changes

aaronlehmann force-pushed the watch-api branch from b5738c0 to e975d7c Compare March 14, 2017 00:35

aaronlehmann mentioned this pull request Mar 17, 2017

API: Events stream #491

Closed

aaronlehmann force-pushed the watch-api branch from e975d7c to 5e5ef6c Compare March 17, 2017 17:44

aaronlehmann force-pushed the watch-api branch from 6d7cd83 to 16f1717 Compare March 17, 2017 19:35

dongluochen reviewed Mar 17, 2017

View reviewed changes

aluzzardi reviewed Mar 18, 2017

View reviewed changes

aaronlehmann force-pushed the watch-api branch from 16f1717 to 59cea93 Compare March 18, 2017 00:53

aaronlehmann mentioned this pull request Mar 21, 2017

Generate data store code #2039

Merged

aaronlehmann force-pushed the watch-api branch from 59cea93 to d32df3e Compare March 21, 2017 17:36

aaronlehmann added 7 commits March 24, 2017 10:50

Add watch API

333f74d

This is the first part of an external store API. Watch returns a stream of new, updated, or removed objects matching the given selectors. Signed-off-by: Aaron Lehmann <[email protected]>

Add IncludeOldObject option to Watch RPC

678fe9e

Signed-off-by: Aaron Lehmann <[email protected]>

Allow multiple watch actions to be specified by ORing them

17dcdf8

Signed-off-by: Aaron Lehmann <[email protected]>

Code-generate WatchMessageEvent

d439fd7

Signed-off-by: Aaron Lehmann <[email protected]>

Code generate event check functions

f51e689

Signed-off-by: Aaron Lehmann <[email protected]>

Code-generate the other type-specific parts of watch API implementation

9472ac4

Signed-off-by: Aaron Lehmann <[email protected]>

aaronlehmann force-pushed the watch-api branch from d32df3e to 9472ac4 Compare March 24, 2017 17:52

aluzzardi reviewed Mar 30, 2017

View reviewed changes

aaronlehmann merged commit bd8224d into moby:master Mar 30, 2017

aaronlehmann deleted the watch-api branch March 30, 2017 01:15

Yshayy mentioned this pull request May 6, 2017

Gateway Timeout when rolling update a scaled service in Docker swarm mode traefik/traefik#1480

Closed

volgorean mentioned this pull request Jun 20, 2017

Support for events mikejmoore/docker-swarm-sdk#1

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

External watch API #2034

External watch API #2034

aaronlehmann commented Mar 14, 2017 •

edited

Loading

stevvooe Mar 14, 2017

aaronlehmann Mar 14, 2017

codecov bot commented Mar 14, 2017 •

edited

Loading

aaronlehmann commented Mar 17, 2017

dongluochen commented Mar 17, 2017

aaronlehmann commented Mar 17, 2017

dongluochen Mar 17, 2017

aaronlehmann Mar 17, 2017

dongluochen Mar 20, 2017

aaronlehmann Mar 21, 2017

dongluochen Mar 21, 2017

aluzzardi Mar 18, 2017

aaronlehmann Mar 18, 2017

aluzzardi Mar 18, 2017

aaronlehmann Mar 18, 2017

dongluochen Mar 18, 2017

aaronlehmann commented Mar 18, 2017

aaronlehmann commented Mar 21, 2017

dongluochen commented Mar 22, 2017

dongluochen commented Mar 24, 2017

aaronlehmann commented Mar 24, 2017

diogomonica commented Mar 24, 2017

aaronlehmann commented Mar 24, 2017

aluzzardi Mar 30, 2017

aaronlehmann Mar 30, 2017

aluzzardi commented Mar 30, 2017

		message StoreObject {
		required WatchSelectors watch_selectors = 1;

External watch API #2034

External watch API #2034

Conversation

aaronlehmann commented Mar 14, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Mar 14, 2017 • edited Loading

Codecov Report

aaronlehmann commented Mar 17, 2017

dongluochen commented Mar 17, 2017

aaronlehmann commented Mar 17, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aaronlehmann commented Mar 18, 2017

aaronlehmann commented Mar 21, 2017

dongluochen commented Mar 22, 2017

dongluochen commented Mar 24, 2017

aaronlehmann commented Mar 24, 2017

diogomonica commented Mar 24, 2017

aaronlehmann commented Mar 24, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aluzzardi commented Mar 30, 2017

aaronlehmann commented Mar 14, 2017 •

edited

Loading

codecov bot commented Mar 14, 2017 •

edited

Loading