Add --watch option to release command

[iandotmartin]

Adds a flag to fluxctl release to observe rollout progress during the release. Shows the numbers of pods desired, up-to-date, available, and outdated.

Note: I wasn't sure if it made sense to add a case somewhere in release_cmd_test for this flag, since there's no change to the release spec. Happy to add one if desired!

Addresses #1424

[squaremo]

I tried this out and it works well! I've made some suggestions for refining it a bit.
It'd be cool if the progress table refreshed in place, but that can come after.

[rndstr]

Had a look and it worked great, thanks @iandotmartin!

I'm thinking about how to display the data. It's quite hard to parse for me now and gauge any progress:

CONTROLLER                CONTAINER  IMAGE            RELEASE   DESIRED  UP-TO-DATE  AVAILABLE  OUTDATED
default:deployment/nginx  nginx      nginx:1.14-perl  updating  3        2           2          2

What about having the column header say Replicas, and the cell value:
<available>/<desired>
if you pass in --verbose it expands
<available>/<desired> (<outdated> outdated, <updated> updated)
?

That would then look like

CONTROLLER                CONTAINER  IMAGE            RELEASE   REPLICAS
default:deployment/nginx  nginx      nginx:1.14-perl  updating  2/3 (2 outdated, 2 updated)

Edit: this could also eventually be displayed in fluxctl list-controller but that can be done separately later

[iandotmartin]

@squaremo @rndstr thanks for the reviews! this is ready for another look when you get a chance :)

[squaremo]

@squaremo @rndstr thanks for the reviews! this is ready for another look when you get a chance :)

👀

[squaremo]

@iandotmartin Please consider cherry-picking ae29a97, which I made to tweak how this works. Main thing it addresses: it needs to wait for the commit to be synced before moving on to show rollout progress, otherwise it will (usually) succeed immediately.

[squaremo]

@rndstr Can you advise on the best format for showing ready and updated pods?

[iandotmartin]

@squaremo cherry-pick'd

[rndstr]

@iandotmartin since we replaced the Ready with Updated, the verbose part should also be changed, bringing it to:
<updated>/<desired> (<outdated> outdated, <ready> ready)

[iandotmartin]

@rndstr yep, seems reasonable! made those updates.

[rndstr]

Nice @iandotmartin, looks good to me.

Submitting release ...
CONTROLLER                STATUS   UPDATES
default:deployment/nginx  success  nginx: nginx:1.14 -> 1.15
Commit pushed:  ac2f3e3
Commit applied: ac2f3e3
Monitoring rollout ...
CONTROLLER                CONTAINER  IMAGE       RELEASE   REPLICAS
default:deployment/nginx  nginx      nginx:1.15  updating  2/3 (2 outdated, 2 ready)

CONTROLLER                CONTAINER  IMAGE       RELEASE   REPLICAS
default:deployment/nginx  nginx      nginx:1.15  updating  2/3 (2 outdated, 2 ready)

CONTROLLER                CONTAINER  IMAGE       RELEASE   REPLICAS
default:deployment/nginx  nginx      nginx:1.15  updating  3/3 (0 outdated, 2 ready)

CONTROLLER                CONTAINER  IMAGE       RELEASE  REPLICAS
default:deployment/nginx  nginx      nginx:1.15  ready    3/3 (0 outdated, 3 ready)

All controllers ready.

[yinzara]

I tried using this with the helm operator and it seems to not work. For example, I have a custom git helm chart called "openapi-web" that I'm deploying to my cluster via the HelmRelease CRD and the helm-operator. When I disable automation and use "fluxctl release --all --update-all-images --watch" I get:
CONTROLLER CONTAINER IMAGE RELEASE REPLICAS
feature-fluxwatch:helmrelease/feature-fluxwatch-openapi-web deployment us.gcr.io/my-project/openapi-web:feature-fluxwatch-0c9ac47 DEPLOYED 0/0

Repeated over and over again and it never completes.

[MilanDasek]

I have exactly same problem. --watch never ends when automation disabled

  annotations:
     flux.weave.works/automated: "false"

BTW:
Is there any way/solution how to be sure whole set of charts/versions was successful?
I mean is it possible to push multiple image releases in one step/commit which will have --watch?