Python 2 Deprecation

[arm4b]

There were some sparks and scattered talks around this topic that slip through the cracks. Let's start an async discussion about Python 2 Deprecation to think more how can we do this.

The Problem

Python 2 is old and reached its EOL. It served us a long time, but today per @blag valid point, Python 2 pulls the project down because of CI issues, our pip dependencies are dropping support for py2 and so it makes maintenance hard. Not to forget about the modern Python 3 features we can't use to keep the codebase py2-compatible.

Removing Python 2 support and relying on Python 3 only is the required thing to do. The only questions are When? and How?.

The Plan

1) Communication

We can't just go and remove python 2 in nearest release, breaking users installations. Don't be that software!
This should be communicated beforehand. Make the blog Announcement, describe the transition plan, expectations and migration path so everyone is on the same page. That's why it's important for @StackStorm/tsc to come up with a conclusion about the desired plan, versions affected, roadmap, what needs to be done, when, who's in game and so on.

2) Deprecation Warnings

See StackStorm/st2#4938 for more info. We'll need a Deprecation Warning in st2 to indicate that Python 2 will be fully removed in st2 v3.X (to be decided) version. As you can imagine, there are many users with old python2 legacy packs they wrote before.

So there are 2 aspects we want to cover:

• OS platforms with py2 installed.
• Packs with py2 codebase.

As faster we can decide on this and implement, - earlier we could remove py2 later.

3) Real Work: EL7 and U18 packages on Python 3

After v3.3.0 CentOS 7 / RHEL 7 and Ubuntu Xenial will be the only platforms with py2 st2. The biggest amount and careful work is here, https://github.com/stackstorm/st2-packages packaging and making sure we prepare python3 virtualenv st2 packages. This task needs a technical research and thinking how to handle it better:

• Can we install python3 as st2.rpm dependency for EL7?
• Can we install python3 as st2.deb dependency for U16?

4) Follow-ups

st2 repository

• code cleanups?
• st2.conf settings which are py2-only?
• st2 repository makefiles, build system, etc?

Remove Python 2 from the CI build matrix

• stackstorm/st2
• st2ci/st2cd workflows
• StackStorm org repos
• StackStorm-Exchange org repos

Puppet, Ansible, Docker, K8s, Vagrant

There might be some simple follow-up tasks to update the affected deployment methods for python 3.
I made sure that Docker/K8s is already covered.

Announce that Python 2 was finally Deprecated

As usual, we can announce it via Release Blog post when the py2 breaking change comes into effect. If someone interested to write more dedicated article about Python 2 lifetime and Python 3 plans, - that would be awesome to say "Good bye Python 2" and celebrate that.

Questions

1. Considering the py2->py3 transition path and deprecation warnings, which release version we want to target for the full py2 removal?

2. A couple of months ago idea about Python 2 Deprecation Working Group was mentioned. This needs to materialize. Who's involved, who wants to drive, help, contribute, work on these tasks? This is a perfect task to get involved and Contribute.

Please share your ideas, opinions, technical insight or voice for participation.
Removing things should be fun!

---

Python 2 Deprecation project to be populated with the tasks: https://github.com/orgs/StackStorm/projects/15

[namachieli]

I think heavy communication is the way to go. Start having warnings in the install that call this out.

[blag]

I think it would be wise to split up the Python 2 discussion into two separate but related parts: the ST2 codebase itself, and packs on Exchange.

ST2 codebase itself

I don't know that we need to have deprecation warnings for Python 2 in our codebase. Everybody who codes in Python 2.7 has been warned about this for a few years now every time they fire up python, so it's not going to be a surprise for anybody.

We also don't have any official corporate sponsors anymore, so I don't feel like we owe it to anybody to continue to support Python 2.7 in future ST2 relases. Python 2.7 is deader than a "just resting" parrot! 😆

Exchange Packs

Exchange packs are a different story. Currently, most packs support both Python 2.7 and Python 3, including tests under both environments. That means that people who are still using ST2 on Python 2.7 might be installing packs using Python 2.7.

Ideally, we should support that until those versions of ST2 are EOL. While we do have a way for packs to advertise support for which Python version they support, I think it's fair to continue to support both Python versions in Exchange packs as long as there are ST2 versions that support Python 2.7.

This means that removing Python 2 code and tests from Exchange packs will be dependent on how quickly we can move to Python 3-only ST2 releases. Once the last version of ST2 that supports Python 2.7 is EOL, then we can start removing Python 2.7 support from Exchange packs and their CI/deploy configs.

The timeline for Exchange packs is therefore much longer than for the ST2 codebase. All the more reason to drop Python 2.7 sooner in ST2 itself, IMO.

[arm4b]

Interesting opinion about the python 2 in Exchange Packs.

I agree that as fast we remove py2 support would be better for the project.

I however disagree we can do it in one release or without a warning because everybody knows py2 is old. This is not the reality.
I believe that communicating the deprecation, including deprecation warnings in st2 is the way to serve the community well. It's based on the the feedback we got previously from community and still getting similar signals.

Hence as earlier we can do this: StackStorm/st2#4938 and start communicating py2 deprecation faster we can really drop it. Hence I believe actionable item here is to add this warning StackStorm/st2#4938. BTW this was initially discussed in May.

[amanda11]

I also have concerns that we are seeing errors being raised where packs work on python 2.7, and not python 3. There was an IMAP one in the last few weeks, before that I noticed ghost2logger only worked on python 2(though that was fixed with ST2 monkey_patch). In last week another raised on mysql. How many packs are heavily being used in python 3? Can we find out from community, or do code analysis (that iter_items problems seems a common one).
Getting an "in your face" deprecation warning to me sounds a good idea.

[guzzijones]

We will have to ensure the OS we target support a package with a python3 named binary. Centos 7, 8. Ubuntu 18, 20. I think. That way we can change all the cmd scripts to use that one. Then we add this package as a dependency. Users need to be warned IMO.

[arm4b]

During the testing of Add python warning if install a pack that only supports python 2 a PR by @amanda11 a few things I noticed:

• A lot of old packs in Stackstorm-exchange don't list the pack_version at all. Ex: https://github.com/StackStorm-Exchange/stackstorm-nagios/blob/master/pack.yaml which means we don't know if they're really py2/py3 compatible.
• It's necessary, but doesn't look enough to show a warning just for pack install scenario as there are may be other corner cases like custom user's packs developed outside of the stackstorm-exchange. They may have similar problem with pack_version absense meaning they don't show a warning on install. We may need to brainstorm other additions to warn users about the Python 2 outside of the packs. Maybe log messages for each st2 service startup if it's started in the py2 environment?
• @blag had an idea to support py2 for StackStorm-Exchange packs for a while after full py2 deprecation. If doing so, we might also need to warn users in a similar way (if pack_version lists py2) that that version will be deprecated as well.

@amanda11 @winem @Sheshagiri @blag WDYT?

[amanda11]

I was wondering whether for packs there was going to need to be a use-python2 flag, to replace the use-puthon3 when we went further with the py2 removal. But then my contrary argument to it was then does that mean people won't move and we still have to CI packs on py2. Though it would make things easier for users.

We could also put in a warning if python versions is not defined.

With the st2ctl start what warning are we giving the users? It isn't their fault that stackstorm on that platform is using py2. So would the message be annoying and unhelpful. Putting in logs would be ok and along the lines of warning them we will move it to py3. But I am not sure what the aim of a warning to them on startup is - it's not like the packs where they can do something about it. We may then get the questions about how to run st2 on Centos 7 with python 3, but as we don't CI it then it wouldn't be a supported variant. Therefore what was the aim of the warning?

[amanda11]

More thoughts. On st2ctl I guess if the warning was along the lines of
"Stackstorm will be deprecating support for python 2 in a near future release. Please make sure all your packs support python 3." might be an idea.
So I might have changed my mind!

[arm4b]

Right, the first message to deliver is about any possible py2 packs. Second is the platform itself running under the python 2 environment which will likely be affected by the breaking py2 -> py3 change in the future releases.
With that warning we could also encourage users migrating U16->U18 or EL7->EL8 platforms respectively.
That sounds like a good "FYI" message so people are aware about the future Python 2 breaking changes and act accordingly.

[amanda11]

Thanks, @armab That makes sense, and clarifies it. So a few things to get across on the py 2 startup then.
On those Py2 systems are we telling them that we will support py3 on their current platform (if its EL7/U16) in the future, or not yet known? Just trying to think of phrasing.

[arm4b]

Not sure how the technical implementation would go, but yes, the future intent is to try to build the st2 packages with the python 3 dependency for both EL7 and U16 platforms.

[arm4b]

So Python 2 deprecation warnings (StackStorm/st2#4938) were implemented by @amanda11 and will be included in the platform since v3.3 release. That includes warnings in:

1. st2 pack install
2. st2ctl
3. st2 services logs on startup

As a @StackStorm/maintainers @StackStorm/contributors team we'll also need to gather and discuss the plans and technical side about Python 2 deprecation.
We'll need someone from @StackStorm/maintainers to lead this effort, write a blog post about Python 2 future plans to keep our community up to date with the Python 2 deprecation and give everyone enough time for migration.

[amanda11]

After "volunteering" to lead this effort, I wanted to update with my understanding of the current status:

• StackStorm supports EL7 and Ubuntu 16.04 with python 2.7.
• StackStorm supports EL8 and Ubuntu 18.04 with python 3.6.
• Python 3.6 is easily able to be installed on EL7 with yum install python3.
• Ubuntu 16.04 ships with python 3.5.2. Ubuntu 16.10 has python 3.6 available via apt-get, but for 16.04 it is available from source or PPA
• There is the option to run packs on EL7 and Ubuntu 16.04 if python3.6 is installed on them, by using the --python3 is available
• Deprecation warnings will appear on EL7 and Ubuntu 16.04 when users have upgraded to StackStorm 3.3

I have a blog post of "where we are", but will wait to publish until I can update to include what our future plans are.

As mentioned above we need to decide what the next steps are and the order. See @blag point above for differences between ST2 and Pack support.

How best to now proceed? Do we want to schedule a meeting to discuss in more detail? Who is interested in attending? Similar time of day to TSC?

Would be good to get some input on agenda for meeting, some suggestions below:

• Do we want to support StackStorm on EL 7 with Python 3.6?
• Do we want to support StackStorm on Ubuntu 16.04 with Python 3, and if so given that it ships with python 3.5, and python 3.6 is only available from source or PPA what python version do we support?
  • NB. Ubuntu 16.04 EOL is April 2021, Ubuntu 20.04 uses python 3.8
• Do we still support packs using python2 once we've migrated to python3, given that older versions of StackStorm will still use it? And if so, then how long for?
• Do have a need to support python2 packs on latest StackStorm once all our supported distributions use python 3? (NB. there is no support for them on latest O/S releases).
• After agreeing where we want to be, then what are the steps and order to get there?
• Can we define a targetted timeline for achieving this?

[nmaludy]

Added my thoughts below:

• Do we want to support StackStorm on EL 7 with Python 3.6?
  • Python 3.8 on EL7 is available in the main repo via the python3 package, should be easy.
• Do we want to support StackStorm on Ubuntu 16.04 with Python 3, and if so given that it ships with python 3.5, and python 3.6 is only available from source or PPA what python version do we support?
  • I would say drop support for 16.04 (since EOL April 2021), instead add support for 20.04, thoughts?
• Do we still support packs using python2 once we've migrated to python3, given that older versions of StackStorm will still use it? And if so, then how long for?
  • Undecided here, but i would say no to python2, we should migrate everything to python3. This will take some effort to ensure all Exchange packs support python3 (should already be done by @blag 's work, but there may be some stragglers).
• Do have a need to support python2 packs on latest StackStorm once all our supported distributions use python 3? (NB. there is no support for them on latest O/S releases).
  • Same answer as above, i say once we migrate to python3, we don't support packs on python2?

[amanda11]

Thanks @nmaludy. Some queries:

• did you mean python 3.8 on EL7 - not 3.6? I am assuming its a typo and you mean 3.6.8? (and I agree, it should be easy to do)
• completely agree on the last point about python2 on latest StackStorm - didn't think we would want to and would just give us a headache - but wanted others opinions.
• I think the problem with Ubuntu 20.04 is that it ships with python 3.8, and there are known issues to resolve before we can use 3.8. That said Ubuntu 16.04 doesn't have python 3.6 easily available. So we have a problem either way to support 2 Ubuntu versions. This one, might be our biggest discussion. I'm not sure who big the 3.8 problem is, but I've seen it mentioned (that said it's a problem we have to fix at some point as we will want to support 20.04).

[nzlosh]

I drew up a comparison between officially supported Python versions and their availability on officially supported OS'. (plus Debian because I'd like to see it included in the list of tested OS'). The colouring is based on how many days before the product hits EOL.

As @blag had already stated elsewhere, Python3.8 is the optimal version to support going forward. If we have Python3.8 support, it will give at least 2 years of official support of Python across Ubuntu, CentOS and Debian.

CentOS 6 and Ubuntu 16.04 support should be dropped now as there's no way forward with these OS'.

[amanda11]

Thanks @nzlosh that's a great table.

One already gone :-) CentOS 6 has already been dropped in the StackStorm 3.3 release.

I'm seeing consensus so far on dropping Ubuntu 16.04 and adding Ubuntu 20.04/Python 3.8 support instead.

Will be interested to see if anyone, has a contrary view?

[arm4b]

That's a good breakdown! Looks like you people covered here all the technical corner cases and dependencies in one topic, eg. what's blocked on what. Great work! 👍

Looking from the community support side, I don't think it's possible to drop the Ubuntu 16.04 support right now, until it reaches its EOL. Turns out however it's the only thing that would block py2 deprecation. I'm not sure if we want to wait with py2 until April 2021 though, py 3.6 from PPA might be an option? I guess timing really depends on team progress around py2 deprecation.

Ubuntu 20.04 and Python 3.8 is great. We can include it anytime as one of the supported platforms as soon as: 1.) Py 3.8 is added to the st2 build matrix, passed, stable, works 2.) New U20 packages are produced and well tested. This work could be done anytime and included in any st2 release as soon as it's ready. @nzlosh want to volunteer starting that work? With the new 3 months st2 release cadence things become easier.

[arm4b]

@StackStorm/maintainers @StackStorm/contributors Let's indeed organize a Python 2 Deprecation Meeting.
We'll announce it so everyone from Community who wanted would be able to join and also discuss.

Same time as TSC meetings. Next week (13 Oct) or Week after (20 Oct) Tuesday? WDYT?

[punkrokk]

Considering no one is paying us to support Ubuntu 16, can we announce we will deprecate it soon rather than have to port python3 there? I know it’s not EOL, but we are a python project mostly and that’s a hard requirement. JP Bourget / @punkrokk

…

On Oct 9, 2020, at 10:18 AM, Eugen C. ***@***.***> wrote:  That's a good breakdown! Looks like you people covered here all the technical corner cases and dependencies in one topic, eg. what's blocked on what. Great work! 👍 Looking from the community support side, I don't think it's possible to drop the Ubuntu 16.04 support right now, until it reaches its EOL. Turns out however it's the only thing that would block py2 deprecation. I'm not sure if we want to wait until April 2021 though, 3.6 from PPA might be an option? I guess timing really depends on team progress around py2 deprecation really. Ubuntu 20.04 and Python 3.8 is great. We can include it anytime as one of the supported platforms as soon as: 1.) Py 3.8 is added to the st2 build matrix, passed, stable, works 2.) New U20 packages are produced and well tested. This work could be done anytime and included in any st2 release. @nzlosh want to volunteer doing that work? — You are receiving this because you are on a team that was mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[cognifloyd]

This is an example PR for what I'd like to do to all exchange packs after StackStorm-Exchange/ci#101 is merged:
StackStorm-Exchange/stackstorm-acos#18

Feedback welcome before I push PRs to all the rest of the packs. I have a branch with these updates prepared for all packs.

Interestingly, once we drop python2.7 tests, the circleci config across packs becomes very consistent for all packs except for the weekly job schedule (Sunday vs Saturday) and some extra circleci jobs in zabbix pack.

[arm4b]

That went a bit excessive.. including Email, Slack, Github notifications.

@cognifloyd To avoid future Github spam in StackStorm-Exchange, instead of opening a bunch of PRs, can you please work with @blag or @nmaludy so they’ll push the changes based on automation script you have directly to master for every pack?

This is pretty much an administrative task (we have st2 bot account for the purpose) and I appreciate your involvement to help the Exchange maintainers.

[cognifloyd]

Yes. Sorry for the spam.
I haven't pushed PRs to all repos yet; I've only committed the changes to my forks. My mistake was including urls like this discussion's in the commit description.
Again, I'm really sorry for spam.

[arm4b]

@cognifloyd NP and thanks for helping there. Let's try to rely on mechanisms we have to minimize the noise while doing the same impact. I'm thinking that the StackStrom-Exchange story might need a bit more prep.

[arm4b]

@StackStorm/tsc If we drop the Python 2 tests in Exchange, I think it's a bit more involved.
Removing Python 2 Tests means de-facto dropping the py2 support. No tests = no support and there is no guarantee or indicator anymore from that point if pack code remains py2-compatible or not as well as no release setting the border line: py2 vs py3.

So removing tests is only start of the story.

We were serious about the approach taken with Python 2 Deprecation for StackStorm, including communication, thinking about user impact, dedicating a release, deprecation warnings and even some API/CLI changes postponed up to 3.5.

Let's be serious about the StackStorm Exchange as well. We could do something close to:

1. Drop Python 2 Tests for all the Exchange Packs (CircleCI) (as discovered by @cognifloyd and @blag).
2. Stop Python 2 support guarantee by removing python_versions: 2 from pack meta.
3. Add the Changelog record for every pack stating Deprecation of the Py2 support.
4. Bump "major" pack versions. Meaning if pack was 1.2.3 we'll release the new 2.0.0 setting the clear border line.
5. Do it in one go via st2 bot account against the master branches for every pack to minimize the noise.
6. Include the update of Dropping py2 in Exchange in the upcoming v3.4.0 Release Announcement (by @blag and @amanda11). That'll be perfectly aligned with the entire st2 py2 removal story.

This way we'll be super clear about what's going on and better balanced with respect to user expectations.

[blag]

No tests = no support

I strongly disagree with this assertion. There's plenty of code in StackStorm, and in StackStorm packs that has no test coverage, but I think we all consider such code still supported.

Ensuring that all supported code is tested is absolutely the ideal, but it is not a rule, and I don't even think it's an expectation.

@armab I like that checklist, but I don't think step 1 has to happen at the same time as step 2. And, if we drop python_versions: 2 at the same time as we release ST2 v3.5, then I don't even think we need to bump the pack version up a major version number.

Most churn in Exchange packs is due to dependencies suddenly causing failures because the dependencies have removed support for Python 2. So in the past we've pinned a lot of the dependencies that are failing. If we remove the Python 2 tests now, and then remove official support for them in May with ST2 v3.5, that's only going to be about 3 months where supported (by us) Python 2 code is exposed but not tested. And users who are still running on Python 2.7 are already aware that they are playing with fire by running on a runtime that has reached EOL, so I don't think the impact is going to be severe. And for whatever impact we do have, honestly, I'm out of empathy for those users at this point. Python 3 has been out for 13 years now. The writing has been on the wall for 13 years. Three more months of us still supporting that code (eg: by not merging in Python 3-only code) but not testing that code is not going to change the risk calculations for anybody still running on Python 2.7.

Edited to add:

Whatever we decide to do, we should consider the cost/benefit of it. We should not go through heroics to keep the Python 2.7 tests running, since they're only going to be useful for the next three months. And keeping support but not tests for Python 2.7 through ST2 v3.5 benefits our users without significantly costing us anything.

I'm don't consider support without tests to be an ideal solution, and I do not advocate that approach in general, but I think in this specific situation it's an acceptable, temporary compromise between keeping our users happy and amount of effort required on our parts.

[cognifloyd]

To be very clear:
Python 2.7 pack tests on StackStorm-Exchange are broken and can't be fixed without significant time and effort.

I already spent several days on it, trying to get it working, and discovering several of the broken pieces. I don't think fixing the python 2.7 tests is worth the effort.