Automatic Man Page Installation Via Setup.py

[BenHays242]

In the README, it states "Since there is no general way to install a manual page for a Python package, one has to do it manually.". I'm not sure this is entirely accurate as the SetupTools.setup function allows for a data_files argument which provides the ability to install files to another location on the system. As this StackOverflow answer states, one could use this to install manpages. I've tested this and it works on my Arch Linux system, although I haven't tested it further than that. I'm happy to make a PR including this feature, but only if you think that this is a feature you would consider in this project.

TLDR: It may be possible to automaticly install man pages via setup.py, is it worthwhile to pursue adding this as a feature?

[BenHays242]

Update: This feature wouldn't work with PIP, but it does with a manual setup.py install. This comes down to the fact that a manual setup.py install requires Super-User, but PIP does not. If we required running PIP as SU that would fix it, but that's not a great solution. We might be able to enable manpage installation via manual install but not on PIP installs but I'm not sure how to achieve that.

[ynikitenko]

Hi Ben!

Thanks a lot for the suggestion! In fact, I read that very same thread on SO, there is even my comment from 3 weeks ago :)

I still think that in general, if there is a system-wide installation, then the system installer is better than pip.

Since there is a package in AUR, why do you want to use pip?

You are right about permissions and user/root installation.

Maybe it would be worth to add the manual to data files? At least the user won't need to download that her/himself! Or if they installed yarsync hastily and went offline, they will have the manual with them :)

What do you think?

[ynikitenko]

I'm not an expert on packaging (I have only two of them, and only one for an OS, not only PyPI). So there might be good suggestions that I don't know yet.
About setup.py: for a system installer it doesn't make difference, whether setup puts the manual in a needed place, or the installation script does it. Without setup it looks more explicit / Pythonic. So for the real installation I would leave it as it is (but to add the manual to PyPI might be a good idea). However, there are many Windows users on PyPI (if we think about them), who don't even have programs to read man-s. This is why in general I think PyPI might be better without mans. Maybe in some future we add a Windows support? Not sure whether rsync will work fine there.

[BenHays242]

Everything you said I 100% agree with, I'm happy to close this issue if you want, but I'm also cool with leaving it here for now. Windows support for man page viewing is actually a really cool idea. Something like https://github.com/halprin/man-viewer except with cross-platform support would make a nice project. It would also alleviate some of the issues regarding UNIX permissions when installing man files as one could view them on a per-user basis.

[ynikitenko]

Thank you. I think this is an important issue, so it may stay until there is a good decision for that (or we completely abandon this).

First, I'd like to write alternatives to man pages. I'd split them into two groups:

• online (readthedocs, etc.) Can be downloaded by user (see no difference with the current man installation, except that it doesn't require root - but one can install a man without root as well).
• installed locally. This doesn't necessarily has to be a man, but an html-file, pdf, etc. This is the preferred solution. It would be good to see some examples of this in established Python programs.
  An excellent variant of this is just Python standard help. Unfortunately, I don't know how to do that without supporting two documentation versions in man and in the Python file. But it may be possible, because our man is simple markdown!

A variant of a local man is a CL option like --long-help. The downside of it is length (will require a pager?) and bad default formatting (for now, argparse don't emphasize any words in the --help output.

Thanks to Mikhail Zelenyy for these suggestions.

Still, I'd like to ask you: why do you prefer pip instead of the package for Arch? I think that use-case may be very important information here.

[ynikitenko]

Man page installation can be

Local

pip run as user. To $HOME/.local/share/man (could not find any official mentioning of this path; does not exist on my system, though I have many programs installed locally, that is .local/bin exists). From SO and AskUbuntu. Downsides:

• Con: the user most likely will have to add this path to his or her MANPATH. (for Ubuntu they claim that it will be searched automatically; for Arch Linux for me it doesn't work)
• Pro: mandb doesn't need to be run as root. I could run mandb as user, and whatis worked fine after that too.

Global

pip run as root. To system manual directory.
Problems: man paths vary, man formats vary ("Debian uses the suffix «.gz», Mandriva (when it existed) used «.lzma», OpenMandriva and Mageia use «.xz»" from setuptools discussion; on my installation I also found just uncompressed files). This problem was solved for bdist_rpm.py, may be worth investigating this way deeper.
Possible solution: man --path rsync will get the path and the suffix for the installed rsync manual.
Pro for this approach: can run mandb as root.

Common problem for local and global: mandb is not necessarily installed. It can be other programs as well, like makedb and makewhatis (posted by a user of Linux Mint, Kubuntu, and Debian). In this regard, the current (manual) installation seems the most robust.

Notes

For local installation one must also take into account virtual envirnoments. If mans are installed to a common directory, they will definitely conflict with each other (they report that users complain about that).

In Python packaging they suggest using

If you want to include additional files, see the documentation for your build backend.

For yarsync we use setuptools. The issue for manual pages is open now.

In 2015 for beetbox they wrote that pip does not install man pages, this was not fixable.

BTW, many people asked ansible to provide a pdf, but it was not done yet. There is also a good suggestion to use both html + pdf (for different devices). Readthedocs would do it great (but one will still have to download the files manually).

ansible uses a special command ansible-doc module_name for this. People complain that this is non-standard.

I think that a reasonable compromise could be:

1. for global installation, find the man path and install there.
2. for local installation, check whether it is virtual environment (if yes, then no man installation or into the package?). If not, can install to the local path. This will require more work, and is lower-priority.

[ynikitenko]

Finally, I think that the standard Python help should be best (but, for user convenience, it must always be identical to the man page). However, I'd also like to add the aspect of uninstall. If a package was installed, it should be uninstalled without leaving any of its files (including mans).

To properly uninstall a data file, some precautions must be made (to be done, if we plan to install for root).

• take care about renamed files (add empty files to keep track of newly created ones with these names).
• provide all created files. In 2015 they suggested overriding the get_outputs method of build_py; not sure that it is needed now (investigation needed).

For virtualenv (and maybe for any other install), it is written in req_install.py:

    Refuses to delete or modify files outside of ``sys.prefix`` -
    thus uninstallation within a virtual environment can only
    modify that virtual environment, even if the virtualenv is
    linked to global site-packages.

Not sure whether this is accurate (what about data files?..), but it can be an issue.

For the moment, I think that Python > help(module) would be best, and it doesn't have this problem (as well as those for virtualenvs, paths, extensions, etc). The only (minor) downside of this is formatting: mans look really good in the terminal (better than Python help). For the online manual and different devices, one may use readthedocs (when/if we add them).

[BenHays242]

I totally agree regarding the preference of using Python help above manpages or a wiki, but wouldn't it be somewhat possible to achieve highlighting in argparse as it's just a matter of using the right control characters when print()-ing. You could also make some modifications to argparse and use that version in this package, or if you don't mind ditching argparse I've heard https://github.com/docopt/docopt is a great alternative with rich text formatting in mind. As for why I prefer PIP over AUR installs, I find relying too much on the AUR to be quite a burden. Plus dealing with packaging relatively small OSS projects to any Linux distro to be a quite time consuming process which could be avoided with something like PIP (Maybe not, looking at some of these links you sent I'm second-guessing the usefulness of PIP) or a polished build/install script.

[BenHays242]

The Oh-My-Zsh project comes to mind regarding really nice install methods that don't require distro-specific packages. They have a one-liner that runs a shell script in their repo that basicly automates the entire process. Below is the one-liner in question:

sh -c "$(curl -fsSL https://raw.github.com/ohmyzsh/ohmyzsh/master/tools/install.sh)"

Having this be the prefered way to install would fix alot of the issues that come up when trying to have wider support for more distros, and it's not like YarSync needs a bunch of libraries, it's just a polished python script at the end of the day, so a shell script could work well if not better as an installation method, just a thought though. This could also allow for easier installation of manpages or any kind of documentation for that matter. Might be overkill though :)

[ynikitenko]

I tested Markdown with Python docstrings. Unfortunately, it seems not supported. Indeed, Python help formats nicely function names, but this seems all (no docstring formatting is done).
Sometimes it is possible, but even then unreliable (https://youtrack.jetbrains.com/issue/PY-40010).
So I think we shall just use readthedocs, so people will have to download documentation themselves (apart from the short help built into the script).

[ynikitenko]

So, I created a readthedocs page: https://yarsync.readthedocs.io/
I think I have to close this issue, because it is not related to the program. If there exists a general way to distribute a man page with a Python package, please share (but I'm afraid that it is in general impossible, or would had been done long ago).
There were also good suggestions here like improving Python help for the script, but I already have a separate issue for interactive help, and it is not related to man page, so they should be made separate issues if needed.
For now I would focus on program's functionality. The documentation is good enough today.