Build ST2 on Ubuntu 16 with Python 3

[blag]

Relevant PRs: StackStorm/st2packaging-dockerfiles#94

[arm4b]

We have the Ubuntu Xenial .deb packages built now.
Next we'll need py 3.6 PPA repository in Xenial packagingtest image to test the st2.deb installation: StackStorm/st2packaging-dockerfiles#95

[arm4b]

The Ubuntu Xenial .deb packages are now ✔️ and built with Python v3.6. I've also added a small test to ensure st2 cli shows python 3.6:

st2 cli version checks
  Command "st2 --version"
    exit_status
      is expected to eq 0
    stdout
      is expected to match /on Python 3.6/

Manual tests for the new py3 U16 packages are still TBD, including upgrade from the py2.

FYI to avoid scripting and modifying other installers I'm also mirroring the PPA python3.6 packages to be installed from our PackageCloud repository. This will ensure smooth py2->py3 transition for the Ubuntu Xenial users before we deprecate U16 in several months in April 2021 and also provide a safety net if 3rd party PPA repository would be deleted at some point.

In terms of timely security updates, https://launchpad.net/~deadsnakes/+archive/ubuntu/ppa has the following disclaimer:

Disclaimer: there's no guarantee of timely updates in case of security problems or other issues. If you want to use them in a security-or-otherwise-critical environment (say, on a production server), you do so at your own risk.

We'll need to provide similar disclaimer when announcing the 3.4.0 which will be another warning for users to migrate to newer OS platform like Ubuntu Bionic. In terms of upgrades, I'll need to periodically check if py3 PPA releases any new package fixes to sync with our PackageCloud.

[amanda11]

I thought we agreed to NOT provide our own python, and tell the user to source their own PPA. So that the upgrade/install steps would tell them to source their own PPA before they did install or upgrade. That way if they have their own PPA, they can use that / or choose must suitable for them.

[arm4b]

Right, I think the agreement was to include the 3rd party deadsnakes PPA in the installer scripts which is the only repository that had python 3.6 for Ubuntu Xenial, otherwise there is no way to install the st2.
Implementation provides more context though with an idea of best migration experience.

If users want to rely on 3rd party PPA - they still can add that with APT Pin-Priority: for their 3-rd party repo. We can provide the recommendations and instructions. On top of that, if deadsnakes PPA releases a new version of py36 package and user had that repo added, - it will be installed automatically as an upgrade instead of ours because of the higher revision number. This looks like a good safety net, if someone wants to go that path.

The main concern was if we build our own python 3.6 package and support it like https://github.com/deadsnakes/python3.6. We don't do this, but instead we mirror what deadsnakes 3rd party PPA repository has.
With that, what are the risks of mirroring the 3rd party PPA py3.6 package from our repo instead of hardcoding 3rd party PPA in installers?

cc @StackStorm/tsc and @StackStorm/contributors

[blag]

With that, what are the risks of mirroring the 3rd party PPA py3.6 package from our repo instead of hardcoding 3rd party PPA in installers?

I'm 100% against this. It provides almost zero benefit for us, and creates a giant maintenance burden/headache for us until at least mid 2021.

The only potential benefit would be that we can review patches to the upstream deadsnakes Python before we pull those changes into our fork. My C skills are incredibly rusty at this point, and I don't think any other ST2 contributors are intimately familiar with the CPython codebase, so none of us would really be able to provide a good code review with any reasonable degree of certainty.

And there are absolutely huge potential pitfalls to blindly copying and patching code - look at the Debian OpenSSL fiasco, where the distro maintainers removed code that the didn't understand, didn't get anybody who knew the code to review their own changes, didn't push their changes back to upstream for review, and created a giant security issue for thousands of Debian users.

So if we're not going to patch anything from deadsnakes, and we aren't going to be able to reasonably review any of their patches, then we're just going to be blindly copying their code and hoping for the best. At that point, we might as well just use their repository anyway.

[amanda11]

I thought to be honest we weren't going to pin any repository, but put the adding of a repo that contains python3.6 as a pre-requisite. As we didn't want to maintain our own repo. We said we could state what repo we had used for our verification (e.g. deadsnakes) but the onus was on the user to decide what repo to use.
So the installer would have a pre-requisite that a repo was added - and we'd have to add that repo as a pre-req in our pipeline, but I didn't think we were adding it to the installers.

So packagingtest would add the repo.

[blag]

I have the same understanding as @amanda11.

[arm4b]

@blag we're not going to patch, support or build python3 packages ourselves. I'd never sign up for that.

I'm saying I'm mirroring what Deadsnakes PPA repository already provides through our PackageCloud repository.

The advantage of mirroring from our PackageCloud is very important:

1. Users have zero friction when upgrading st2 v3.3 U16 py2 -> st2 v3.4 U18 py3. The upgrade is automatic with zero context, pre-req, scripting, etc.

^ That's the main point which will reduce the support burden and issues happening in community.
We don't need to include additional py3 installation scripts in installers.
We don't need to provide migration docs for existing installations (which, let's be realistic, very few people read).
In terms of installation it also reduces the amount of moving pieces: adding 3rd party PPA, another point of failure, downtime or possibility if Deadsnakes will be deleted.

[arm4b]

@amanda11 @blag RE agreement of not adding py3 PPA in installers. Hm, let me check the history. I might be smoking something then.

[arm4b]

@amanda11 @blag I'm re-reading StackStorm/community#53 (comment) and it indeed states the:

Add support for U16 and Python 3.6 with customer's own supplied PPA

OK, looks like I'm wrong with the agreement to provide py3.6 via Deadnakes 3rd party PPA by default. I assume I've lost the track then in the discussion or indeed smoked something. Sorry for that.

---

Now after digging into implementation details and speaking technically. The agreement is that we just provide a documentation that users have to find their own python 3.6 repo for Ubuntu 16.04.

1. There is only 1 APT realistic repository out there with python 3 for Xenial and it's Deadsnakes PPA.
   The package name we depend on in Requirements is not python v3.6, but python3.6, - very unique. If we build the st2.deb relying on this specific package, there are little to no alternatives for users and they have to use that specific python3.6.deb as we've built with and tested with.

2. Next, If there is no U16 py3 repo codified in the installers, how the e2e tests, automation and deployments would work? Do we kind of install ppa via e2e workflows instead of including in installer? This would be pretty much faking the Ubuntu 16.04 installation/support.

3. Then, if we don't add U16 py3 in the installers, how do we handle the support burden in Slack, Forums for users having these installation and upgrade issues? Just adding st2docs or warnings won't help much, realistically speaking.

4. Because of the installation issues, if community users provides a PR to add the Py3 repository to curl|bash installer, Ansible, etc, will we refuse those PRs?

And looks like I voted for that option. This doesn't fit the picture of "supporting U16". Sounds like I voted then for throwing a pig over a fence 🤦

[amanda11]

Given the warning on the deadsnakes PPA then I would suggest that we want people to consider if PY3 on Ubuntu 16 is for them. If it is, then they can install the PPA and accept the warnings, or they upgrade to Ubuntu 18.
If the argument is that they won't read the docs about it, then that problem exists with putting a warning into the docs about python3 on Ubuntu 16. So if it gets taken care of by the installer, then no-one will see that warning, and therefore not understand the consequences.

Also on the upgrade path, then I'd state that the upgrade process is already documented as manual steps that can vary between releases - and therefore I don't see an issue with it being in step specific to that upgrade, just as we had for upgrading mongodb etc.

[blag]

No worries, nobody's perfect!

Next, If there is no U16 py3 repo codified in the installers, how the e2e tests, automation and deployments would work? Do we kind of install ppa via e2e workflows instead of including in installer? This would be pretty much faking the Ubuntu 16.04 installation/support.

That's exactly what I did in StackStorm/st2cd#446, which is installed on st2cicd (I thought I mentioned this in Slack, but I can't find it right now).

I don't think many people are running on Ubuntu 16.04 anymore, and if they are then this is an excellent reason to either not upgrade StackStorm, or to also upgrade their OS to Ubuntu 18.04. If they really want to stay on 16.04, then they can read and heed the upgrade instructions.

^{Or we can drop Ubuntu 16.04 support entirely.}

[arm4b]

OK, based on @StackStorm/tsc discussion, the solution is to pause the curl|bash st2 installation waiting for the user input when Ubuntu16.04 has no available py3 package candidate.

• yes will install the Deadsnakes PPA, no will exit the installation process.
• there should be a Warning that users are about to add 3rd party py3 PPA on their own security risk and recommend to migrate to U18.
• We'll need a CLI flag --maybe-yes (naming TBD) to allow automatic installs.
• Same for Ansible/Puppet install.
• Ubuntu 16.04 py2 -> py3 upgrades, - users have to rely on a manual Upgrade instructions we'll provide in st2docs. Recommend 3rd party Deadsnakes PPA with a security disclaimer.

That'll give us acceptable balance between security/usability and we could e2e and CI things as well.

[arm4b]

Merging to fix the CI ✔️
Will ship the discussed ^^ curl|bash installer yes/no changes in another PR.