Cannot undo ignore annotation

[tamirms]

I added an flux.weave.works/ignore: "true" annotation to some of my kubernetes deployments. In a subsequent commit to my kubernetes repo, I removed the annotations. However, flux still seems to ignore my deployments. I've tried killing the flux and flux memcached pods but that still hasn't fixed the problem. I've also tried renaming my manifest file.

some more info from slack:

in the weave flux agent logs I see
```ts=2018-07-06T17:41:26.352552842Z caller=sync.go:70 component=sync-loop resource=default:deployment/pb-production-accounts-deployment ignore=apply```
so for some reason the ignore tag is stuck to the deployment
I have triple checked our manifest file
and there are no annotations at all

stefan [7:47 PM]
what version are you running?

tamir [7:47 PM]
how can I find that info?

stefan [7:47 PM]
`kubectl get deployment/flux -oyaml`
check the image tag in the output (edited)

tamir [7:54 PM]
I ran `kubectl get deployment/weave-flux-agent -o yaml --namespace=weave`
@stefan the image tag is `quay.io/weaveworks/flux:1.4.2`

stefan [7:56 PM]
that's the latest version, it should work... maybe @mbridgen has any idea how to diagnose this

tamir [8:06 PM]
I’ve tried reinstalling as well
doesn’t seem to do anything

stefan [8:07 PM]
try renaming the deployment file

tamir [8:07 PM]
ok

tamir [8:14 PM]
it didn’t work
`ts=2018-07-06T18:11:55.108793942Z caller=sync.go:70 component=sync-loop resource=default:deployment/pb-production-accounts-deployment ignore=apply`
still shows up in the logs (edited)
I guess I’ll have to do things manually
@mbridgen @stefan let me know if you require any more info from me to diagnose this issue

stefan [8:16 PM]
one more try, can you please delete the flux pod and the memcache one

tamir [8:17 PM]
ok

tamir [8:29 PM]
still didn’t work :disappointed:
is there some volume memcached is using to persist its state?
maybe I can delete that as well?

stefan [8:37 PM]
Nop, my guess is that the git pull that Flux does gets somehow an older version. Can you check in Weave Cloud that you have the right branch

tamir [8:41 PM]
i see this
`ts=2018-07-06T18:37:57.633720149Z caller=loop.go:102 component=sync-loop event=refreshed [email protected]:... branch=master HEAD=5126c700b0ea20a1aa3677f9bcf700ad4b4f6200`
the git commit hash is correct
everything looks correctly configured in weave cloud as well

stefan [9:02 PM]
Please make an issue on GitHub with these logs

[squaremo]

Do the resources in the cluster carry the annotation? You will need to remove it from those, as well, because fluxd will ignore something if either the manifest, or the resource itself, has the annotation.

kubectl annotate <resource> "flux.weave.works/ignore"-

[tamirms]

@squaremo ah that looks like the cause! so if I want to undo the ignore annotation I should update the yaml configuration in my git repo and also manually run kubectl annotate <resource> "flux.weave.works/ignore"- ?

I would expect that updating the yaml configuration and pushing that to git would also remove the annotation from the resource itself. Also, it seems that running kubectl apply -f ... on the updated configuration does not remove the annotation.. is that to be expected?

[squaremo]

so if I want to undo the ignore annotation I should update the yaml configuration in my git repo and also manually run kubectl annotate "flux.weave.works/ignore"- ?

Yes, that should do it. If you want to use ignore in future, it's probably better to just annotate the resource in the cluster. (You might want to annotate the manifest if you didn't want something created in the first place, though.)

I would expect that updating the yaml configuration and pushing that to git would also remove the annotation from the resource itself.

Well .. it's ignoring it ..
Though you might ask "why does it look at the annotations of both the resource in the cluster, and the manifest in git"?

The answer is that the ignore mechanism tries to account for different scenarios, and there's not always a file corresponding to a resource, or a resource corresponding to a file. There was quite a discussion about it, back in #469 (comment).

Also, it seems that running kubectl apply -f ... on the updated configuration does not remove the annotation.. is that to be expected?

Yes; the way kubectl apply merges things can sometimes mean fields that are not mentioned in the update are not removed in the merged resource.

[tamirms]

ok if this is expected behavior feel free to close the issue

[squaremo]

Sorry if this held you up @tamirms. It's still a somewhat experimental feature, and I don't think it yet hits the simple v useful sweet spot.

[stefanprodan]

@dholbach please add the instructions on how to remove the annotation with kubectl to #1210 removing the annotation from Git is not going to work.

[arash-bizcover]

I put fluxcd.io/ignore: "true" on annotations of a Deployment in cluster,
Flux doesn't ignore it; changing replicas and it gets reverted 😶

Flux v0.26.0 (controller:v0.20.0)

[dholbach]

@arash-bizcover This issue is about Flux v1 (Flux Legacy) - you should have a look at https://fluxcd.io/docs/ (and https://github.com/fluxcd/flux2) for current versions of Flux instead.