consider getting rid of packages/binaries built by pyinstaller (windows exe and mac pkg)

[efiop]

We've been running into issues with pyinstaller-built binaries for a long time and now I wonder if we should abandon them for good.

Mac: we have brew, pip and conda, doesn't seem like there is any need for our own pkg installer. We can't really make pkg installer install non-binary package for now, as we depend on python and mac has py2, which is soon to be dead. Supporting py2 only for that is not an option as well, since it causes a lot more issues than pyinstaller, so we'll drop py2 a bit before or right at Jan 1 2020. Considering how popular and powerful brew, pip and conda are, feels like we can just drop pkg for good.

Win: we have pip and conda, feels like something like chocolatey package would be a nice addition, that would eliminate the need for our own exe installer, but I haven't researched it enough. Also can't really make our current exe install non-binary, as there is no default python to rely on :slight_frown: Seems like conda is de-facto the standard for DS on windows, so maybe we could also simply drop the exe.

Linux: we have pip and conda. We also have deb and rpm packages and repos, which are currently packing binaries, but we could totally make it python-dependendent. That would take a bit more work to convert dep python packages, but it seems like a good idea in the long run, if any of the deb/rpm distros ever decide to include dvc into default repos.

From https://discordapp.com/channels/485586884165107732/565699007037571084/642407396790239254

[efiop]

If there are any users that need these, please let us know here in the comments. 🙂

[efiop]

Some stats on our pkg and exe downloads: https://www.somsubhra.com/github-release-stats/?username=iterative&repository=dvc Some of those are from our tests. So not many downloads for windows and mac installers. deb/pkg are usually served from our s3 repos, so those don't count.

[ghost]

Python is not that hard to get on Windows: https://blog.adafruit.com/2019/06/04/new-guide-python-is-in-the-latest-build-of-windows-10-adafruitlearningsystem-python-adafruit-win10-windows-adafruit/

You can search it on the store, without going to Python.org 👀

https://www.microsoft.com/en-us/p/python-37/9nj46sx7x90p

[efiop]

@MrOutis Still not as simple as current installer. Maybe there is a better way to package this.

[efiop]

Not finding any good alternative :( Looks like the options for distributing python app are the same as they were 2 years ago. Looks like we will simply have to suck it up and improve our testing to catch those issues early. 🙁

[efiop]

Things like awscli are also still using pyinstaller, even though they run into similar issues from time to time. Looks like there is no other way for now. Closing.

[ghost]

@efiop , I think you are overestimating the installation task; right now is as easy as searching python on the store, click download, pip install dvc and you are done :)

---

It is not like drawing an owl or something

[efiop]

@MrOutis That's a good point! Though awscli, mercurial and others do provide windows/mac installers to make it a1 stop shop, instead of making you install python and run pip. To me it seems like our users are advanced enough to do that, but it might not be a valid excuse to raise the bar for the newbies. Need to learn more about why people are even using those. Will keep this opened for now, maybe someone will share their thoughts (i'll also try to actively ask users about that).

[ghost]

looks like the reason behind aws-cli providing Windows installers is the lack of Python providers from the OS: aws/aws-cli#2501 (comment)

And from their web page, looks like they are going with pip for OSX: https://aws.amazon.com/cli/

[efiop]

@MrOutis Yeah, but default python on mac is not very good too 🙁 It is an ancient python 2, which we will drop in 2 months, so can't rely on it 🙁

[ghost]

@efiop , true that 😞

I read that they are discouraging users to rely on python 2: https://developer.apple.com/documentation/macos_release_notes/macos_catalina_10_15_release_notes

[casperdcl]

Since you asked @MrOutis - my opinion on this issue is the same thing I always say: who needs users when you have devs?

On a serious note I also go through similar pain packaging executables bundling a python interpreter for windows. Don't think there's an easy way around it. Recommending anaconda and/or chocolatey and/or docker may work. Speaking of which... New issue time.

[efiop]

For the record: reasons why users are using bin packages:

#2749 (comment)

I initially used it because it seemed the easiest way to provide single dvc version for several Unix users

#2754 (comment)

I installed DVC via scoop which uses the .exe file instead of pip or conda.

[nightscape]

If I may chime in here as a Windows user:
I've installed dvc via scoop, which imo is the nicest way to install stuff on Windows.
Here is the package definition for dvc.
It doesn't require .exe files, but usually some kind of archive that can be downloaded and unzipped (e.g. to ~/scoop/apps/dvc/0.66.9). It then sets paths, creates shims, etc. in order to put things on the path.
Afaik the innosetup part is rather an annoyance in that case, because innosetup packages can't just be unpacked, but have to be installed, with potentially unknown side effects.
So, from scoop perspective having some archive which contains everything needed would be the preferred option.

[casperdcl]

Hmm does that scoop repo link get an automated PR every single time one of the supported packages gets an update? Seems like a nightmare to manage. conda-forge meanwhile delegates merging in automated PRs to maintainers of the individual packages, though that comes with other issues like conda-forge/conda-forge.github.io#908

I'll also post another vaguely related comic with the excuse that I didn't post the first one so can't be accused of starting this...

[nightscape]

@casperdcl afaik each scoop bucket contains a script that updates all dependencies. That script is run automatically every hour or so and the result automatically committed and pushed to Github.
Imo the approach is really great. I've submitted quite a few package declarations now, it really takes only a few minutes to provide them and in return I get free package updates for live 😄

[ghost]

I'm still in favor of deprecating pyinstaller, although, we can wait until we drop support for Python 2)

@nightscape , would requiring a Windows user to install Python (either by a package manager, store, or via the python.org webpage) and then use pip to manage (install/upgrade/etc.) DVC would be a significant downgrade on the user experience?

There's also the option to use miniconda

[efiop]

@MrOutis I also don't like pyinstaller that much, but sadly I don't see any other option right now. Dropping py2 wouldn't change it. Our best choice it to be careful and patient here 🙂It seems like chocolatey is also a very good bet, I've created #2833 for it, seems to be pretty easy to install dvc with choco install dvc(it will install everything automatically, as long as you have choco installed).

[nightscape]

@MrOutis I haven't found any scoop packages that install via pip (nteract installs ipykernel via pip, but still relies on a .exe file to do the main installation), but of course users could just manage dvc outside of scoop.

[casperdcl]

On this topic, what is the best possible way to install dvc? My guess is:

1. preferably use Ubuntu 18.04
2. install conda
3. conda create -n dvc python=3 dvc[all,ssh_gssapi,hdfs]
4. mkdir -p ~/.local/bin/
5. possibly make sure the dir above is added to $PATH in ~/.profile or ~/.bashrc
6. cat '#!'"$CONDA_PREFIX"/envs/dvc/bin/dvc > ~/.local/bin/dvc
7. chmod +x ~/.local/bin/dvc

And to update

conda update -n dvc --all

[efiop]

For the record: we now have a chocolatey package for windows https://chocolatey.org/packages/dvc . Waiting for moderators and some final testing.

[casperdcl]

We also have a snap package (https://snapcraft.io/dvc, not fully working due to restricted privileges/permissions) and pending request for permissions & inclusion in the snap store (https://forum.snapcraft.io/t/classic-confinement-request-for-dvc/14124).

In the meantime we have our own GitHub-hosted release (https://github.com/iterative/dvc/releases -> https://github.com/iterative/dvc/releases/download/0.72.0/dvc_0.72.0_amd64.snap and install instructions https://github.com/iterative/dvc/#snap-snapcraft) which should work on anything from Arch, CentOS, Debian, Fedora, Kubuntu, Manjaro, Mint, openSUSE, RHEL, and of course Ubuntu amongst others...

[ghost]

By the way, heroku-cli is using snap for Linux-based distros: https://devcenter.heroku.com/articles/heroku-cli#download-and-install

[shcheklein]

and aws-cli is snap distributed as well:

that's what ubuntu recommends if command is not found.

sudo snap install aws-cli # version 1.16.266, or

[efiop]

We are now signing our binaries and they are still useful. Pyinstaller is causing some problems from time to time (usually just because of a missing hook), but the benefits are still worth keeping it around. Closing.