fix status checks for multiple modules

[jashandeep-sohi]

Issue:

If you have workloads in different namespaces managed in different modules, Skaffold will only run status checks on the first module it encounters. This can be a problem when you have dependencies between the modules, where workloads in one must be healthy before moving onto the next (e.g. CRD operators).

For example,

apiVersion: skaffold/v4beta8
kind: Config
metadata:
  name: a
manifests:
  kustomize:
    paths:
      - ./deploy-a.yaml
deploy:
  kubectl:
    flags:
      apply:
        -  --server-side
---
apiVersion: skaffold/v4beta8
kind: Config
metadata:
  name: b
manifests:
  kustomize:
    paths:
      - ./deploy-b.yaml
deploy:
  kubectl:
    flags:
      apply:
        -  --server-side

Where deploy-a.yaml looks something like:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: a
  namespace: a
spec:
  template:
    spec:
      containers:
        - name: default
          image: ...
          command: ["sleep", "inf"]
          livenessProbe:
            exec:
              command: ["true"]
            initialDelaySeconds: 15
            periodSeconds: 5

And deploy-b.yaml looks something like:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: b
  namespace: b
spec:
  template:
    spec:
      containers:
        - name: default
          image: ...
          command: ["sleep", "inf"]
          livenessProbe:
            exec:
              command: ["true"]
            initialDelaySeconds: 15
            periodSeconds: 5

Notes:

• There's a Deployer created per config/module
• A Deployer holds a reference to a list of namespaces to monitor that it updates as it deploys resources
• A reference to this list is also passed to the Status Monitor that it uses to figure our what resources to monitor
• But the Status Monitor is deduped per kubecontext, which leads to a mismatch between the reference to the namespace list for subsequent modules (assuming they use the same kubecontext)
• There should be a Status Monitor per Deployer
• Dedup on the config name (those are unique right?)

Also added a Flake to this project, making it easy to install/run Skaffold with Nix:

nix run github:<org>/skaffold/<rev-or-ref>