Mitogen: deprecate the use of mitogen and remove coverage from CI

[cristicalin]

What type of PR is this?

/kind cleanup

What this PR does / why we need it:
Mitogen upsteam is not seeing any new releases and our continued support for it hinders adoption of newer Ansible versions that would in turn give us support for newer Enterprise Linux versions like Rocky. This PR deprecates the use of mitogen in our CI and explicitly marks it as deprecated in the docs. The install playbook as well as the 3 workarounds are still left in place and I plan to remove them after we tag 2.18.

Related discussion in #7887 and #8125

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

Mitogen: support for the mitogen playbook accelerator is now deprecated in preparation of ansible upgrades, please clean up your playbooks that depend on it.

[oomichi]

The reason is written well on the pull request message, thanks for doing this.

/lgtm

[floryut]

Maybe don't be that specific ? The supported playbook and documentation will be removed in a future kubespray release for example ? WDYT ?

[cristicalin]

I wanted the message to be loud and clear but yes, it's better to temper it down.

[cristicalin]

I changed the wording, also better English language, but kept the warning note.

[oomichi]

/lgtm

[floryut]

Thanks and agreed with Mitogen becoming harder to support as no real activity on their end 😢

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cristicalin, floryut

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [floryut]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[Payback159]

Hi guys, I understand that maintenance is more difficult to impossible if nothing happens on the part of upstream mitogen but I am concerned that this will slow down the execution time of Kubespray.

How do you guys see this or maybe you have already tested a run without mitogen?

[cristicalin]

@Payback159 see #7887 (comment) for some comparisons between ansible with ssh pipelining (current kubespray default) and mitogen, in our tests mitogen with modern ansible is actually slower.

That said, the current deprecation does not remove the capability to use mitogen and if you source kubespray in your own playbooks you can always re-enable mitogen and submit issues and/or fixes to kubespray if you encounter issues.

Secondly, I see there was a recent merge in mitogen master branch to make it work with modern ansible and should a new tag be added and released, we will happily update it in our playbooks and give up on the plan to fully drop it.

To putting it back in CI, we expect mitogen upstream to have a firm commitment to maintain their project or have someone more willing to fork and maintain it.

[maxpain]

in our tests mitogen with modern ansible is actually slower.

In our case, upgrade-cluster.yml on a cluster with 2 nodes without mitogen takes 1 hour 20 minutes.
With mitogen about 20 minutes.

The difference is noticeable when you have a high latency between ansible host and target nodes, in my case I have latency 270ms (from Russia, Moscow to Buenos Aires, Argentina).

[maxpain]

By the way, Mitogen 0.3.1 release is here with Ansible 3, 4 and 5 support.

Is it possible to roll back this MR?

[cristicalin]

@maxpain this PR disables the CI coverage so it does not interfere with you deploying mitogen via the embedded mitogen.yml playbook. The only thing you need to continue using mitogen is to modify ansible.cfg according to the docs.

I'm hesitant to roll this change back since a minor release is not quite proof of long term commitment. We should continue to keep an eye out for mitogen evolution and reconsider adding CI coverage back after it has proven its capability to keep up to date with changes.

I think we can still merge fixes for corner cases tripped by the use of mitogen.