DNS update issues with repeated attempts to add the same node

[verdurin]

Owing to some local issues, a recent attempt to add a compute node to a cluster failed at the stage of provisioning keys.

I found that when I destroyed that node then re-added after the local issues had been fixed, the DNS for the new instance had the IP of the previous attempt, so logins to that node failed.

I had expected the DNS entry in IPA to be destroyed alongside the node.

[verdurin]

Hmm, this may have already been fixed in ComputeCanada/puppet-magic_castle#69

[cmd-ntrf]

Which version of Magic Castle are you using? The fix was released with 8.3.

If you have a running cluster built with an older version of MC, you can cherry pick the fix and apply it to /etc/puppetlabs/code/environments/production/site/profile/manifests/freeipa.pp on mgmt1. Next time you add a node, it will be correctly registered even if a node with the same name was registered previously.

[verdurin]

Not sure of the exact version - it's been running since late August (users very happy).

Will look at the cherry pick fix.

[verdurin]

Hmm, that hasn't worked. While the DNS record resolves, and the new node is picked up via Consul and hence Slurm, DNS resolution of the new node still reflects an old IP.

[cmd-ntrf]

For clarification: was the new node added after the cherry-pick of the fix?

While not ideal, you can always edit the DNS records using the FreeIPA web interface at : https://ipa.nameofyourcluster.domain.tld/

[verdurin]

Yes, this is after the fix.

Was aware of the ability to edit the records, thanks, just hoping to avoid it.

I've just tried a name new to this cluster, and neither host nor DNS records have been created for it. Might be missing some other fixes?

[verdurin]

In the end I found that only re-enrolling the newly-added instance fixed all the issues, after editing the DNS record... (e.g. Slurm jobs failed owing to user-lookup errors)

[cmd-ntrf]

I discover this morning that sometime the ipa-client-install command fails to update the DNS, but still return 0. I have created an issue here : ComputeCanada/puppet-magic_castle#79

It could be related to your problem. I will probably need to add some check in puppet that the record A is created and corresponds to the ip address to avoid these kind of issues in the future.

[verdurin]

Yes, I saw that issue, thanks.

If I have time tonight I'll try again and see what's in the logs.

[cmd-ntrf]

Now that #79 is fixed, I will close this.

Feel free to re-open if the problem persists in coming versions.