Add await logic for DaemonSets

[RichardWLaub]

We are deploying the sysdig helm chart that creates a daemonSet with a rollingUpdate updateStrategy. We immediately run tests to make sure the pods are running correctly afterward. With deployments and services our tests always pass because pulumi waits to verify that those kinds deployed successfully. Our daemonSet tests sometimes fail because there is no await logic for daemonsets.

Simple repro:

import * as k8s from '@pulumi/kubernetes';

new k8s.helm.v2.Chart('simple-nginx-local', {
  path: 'ds-chart',
  values: {
    image: {
      tag: '1.16'
    }
  }
});

With a chart with template ds-chart/templates/daemonset.yaml:

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: test
spec:
  selector:
    matchLabels:
      name: nginx-ds
  template:
    metadata:
      labels:
        name: nginx-ds
      name: nginx-ds
    spec:
      containers:
        - name: {{ .Chart.Name }}
          image: "nginx:{{ .Values.image.tag }}"
          imagePullPolicy: {{ .Values.image.pullPolicy }}
  updateStrategy:
    rollingUpdate:
      maxUnavailable: 1
    type: RollingUpdate

Toggle the tag above to 1.17 to see that the pulumi program finishes before the new pods are ready.

[casey-robertson]

Raising awareness. We had a sysdig outage that could have been mitigated sooner if this was in place.

[blampe]

@RichardWLaub @casey-robertson @rquitales I wanted to share the current approach I'm taking with this in case you have any concerns or suggestions.

For DaemonSets with a RollingUpdate strategy, we will essentially follow the same behavior as kubectl rollout status. Pulumi will wait until the rollout status is ready unless the DS has a skipAwait annotation.

It's less clear how to handle the OnDelete update strategy. For background, we currently handle StatefulSets with OnDelete update strategies by waiting for all pods to be manually removed (#2473); if the user doesn't want this behavior they must annotate the OnDelete StatefulSet with skipAwait.

I'm preserving this behavior for OnDelete DaemonSets for consistently, although I see an argument that Pulumi should not wait for this update strategy at all.