[WIP] Setuptools packaging improvements - make each st2 component packagable and installable via package

[Kami]

This pull requests builds on top of of #1647 and #1668.

I spent a lot of time finishing, polishing and testing all the changes from #1668 locally. Everything worked and looked fine so I decided to merge it into master. Sadly this turned out to be a mistake - there were many, many issues with packaging and other workflows used by our CI.

After 3-4 hours of digging in and fixing issues, I still haven't managed to fix all of them so I decided to revert changes in the master and unbreak it. I believe that's better than trying to rush to whole thing. Next step will be to address all the issues and run all the packaging and related workflows on this branch and only merge it into master when everything passes.

I will work with / ask @DoriftoShoes for help with running everything on a particular branch (not sure if we are there yet, might require some changes).

The biggest offenders currently are:

• Making sure we use pip >= 6.0.0 everywhere. I believe I managed to fix this by using virtualenv everywhere, but there might still be issues somewhere.
• Getting rid of the "recursive-include" abuse in the MANIFEST.in and replacing it with collecting all the code packages inside the setup.pyfiles.

This branch includes the following changes and improvements:

1. Add new dist_utils.py module which is copied over to each component during build step and used inside setup.py instead of copy and pasting code over in setup.py.
2. Update all the setup.py files to include license and url attribute.
3. Add long_description attribute to each setup.py file. This attribute is generated by combining together README.rest and CHANGELOG.rst file.
4. Port README from Markdown to rst (needed because of 3).
5. Add new sdist Makefilet target which generates .tar.gz and wheel archive for each component.
6. Update setup.py files to use __version attribute from each package init.py file. For now we simply copy over init.py from st2common. This way we don't break out existing release / build workflow, but eventually we should move to per-package version.
7. Removed unused dependencies: flask, flask_json. This was actually done by @dennybaa, but I verified the changes and I'm pointing it out there to make it easier to track.
8. Sort the requirements before writing them in the requirements.txt file. This way we make sure generated requirements files are always sorted the same.

TODO:

• Verify it doesn't break our current release workflows
• Add end to end tests
• Add new workflow steps for generating Python package distributions and publishing to to PyPi (not needed right now)

[Kami]

@dennybaa See the pull request description above. I had a really bad time with it.

There is a lot more of work which needs to be done making sure everything works end to end. Among other things, we need to get ride of recursive-include abuse in MANIFEST.in - using "*" means it will also try to include *.pyc and other files which are created while running setup.py dist command which will cause those commands to fail.

As noted in your other PR, we should explicitly collect Python package and module files inside setup.py and only include LICENSE, templates and files like that inside MANIFEST.in. Abusing recursive-include * is a bad idea since it results in all the files generated during dist phase also being included. You can exclude it with prune directive in MANIFEST, but that's a hack and we simply shouldn't abuse recursive-include.

[Kami]

I managed to get make debs and make rpms work fine locally now.

Before merging anything, I would like to run all the workflows on this branch and make sure everything works. @DoriftoShoes what's the easiest way to do this right now?

[DoriftoShoes]

make debs and make rpms will loop through all the components and build packages for them but it will not fail if one package does not build. This is why we have the package verification step in the build workflows. You have to look in ~/debbuild and ~/rpmbuild on the systems where you ran make to see that all the packages got created.

[Kami]

@DoriftoShoes I tried building the packages locally in the VM (I installed all the dependencies) and everything worked fine.

I would like to run all the workflows which run when we merged a branch into master. I did merge branch into master yesterday after everything worked locally, but things exploded on the CI so I would like to make sure everything works before merging the branch and breaking everything (I needed to do a force revert yesterday).