Move to embed and update more dependencies #2424

phillxnet · 2022-11-10T19:21:09Z

During the development involved in moving us from our v3 (CentOS base) to our v4 "Built on openSUSE" we partly transitioned to using some OS system python libraries. It is now required that we transition back to including as many dependencies as is practical.

Three reasons for this proposed direction/requirement:

It is the industry standard for Python apps - however we are a little more than an app given our close integration with the OS, but still.
We are still dependant on some older legacy Python modules, at least until: move to python 3 #1877 is resolved. See also Add Leap 15.4 profiles rockstor-installer#123.
We plan to carry our own Python interpreter in newer major releases so that we are no longer bound by our base OS and it's official Python version.

Current package (rpm) defined dependencies directly related to Python; in the context of the above referenced issues:

via:
rpm -qpR rockstor-4.5.0-0.x86_64.rpm | grep "python"

The text was updated successfully, but these errors were encountered:

phillxnet · 2022-11-10T19:36:16Z

Re: current system package: python-dbus-python

In our Leap 15.3 we had:

zypper info python-dbus-python
Retrieving repository 'Rockstor-Testing' metadata ...........................................................................................................................[done]
Building repository 'Rockstor-Testing' cache ................................................................................................................................[done]
Retrieving repository 'Update repository with updates from SUSE Linux Enterprise 15' metadata ...............................................................................[done]
Building repository 'Update repository with updates from SUSE Linux Enterprise 15' cache ....................................................................................[done]
Loading repository data...
Reading installed packages...


package 'python-dbus-python' not found.

Information for srcpackage python-dbus-python:
----------------------------------------------
Repository            : Update repository with updates from SUSE Linux Enterprise 15
Name                  : python-dbus-python
Version               : 1.2.16-6.3.1
Arch                  : noarch
Vendor                : SUSE LLC <https://www.suse.com/>
Summary               : Python bindings for D-Bus
Description           : 
    D-Bus python bindings for use with python programs.
Builds binary package : 
    S | Name                            | Version
    --+---------------------------------+-------------
      | python-dbus-python-common-devel | 1.2.16-6.3.1
    i | python2-dbus-python             | 1.2.16-6.3.1
      | python2-dbus-python-devel       | 1.2.16-6.3.1
    i | python3-dbus-python             | 1.2.16-6.3.1
      | python3-dbus-python-devel       | 1.2.16-6.3.1

So it was a meta-package for, in our case: python2-dbus-python | 1.2.16-6.3.1

There is on PyPi.org:
https://pypi.org/project/dbus-python/
With a changelog:
https://dbus.freedesktop.org/doc/dbus-python/news.html

dbus Python Bindings

1.2.18 (2021-07-20)

Dependencies:

Python 2 reached end-of-life on 2020-01-01. A future version of dbus-python is likely to remove Python 2 support.

1.3.0 (2022-09-06)

Dependencies:

Python 3, version 3.5 or later, is required. An upstream-supported version (currently 3.7 or later) is strongly recommended, and a future version of dbus-python is likely to require Python 3.7 or later. Python 2 is no longer supported.

New dev dependency

New development system requirement for building the PyPi dbus-python v1.2.18

checking for dbus-1 >= 1.8... no
  configure: error: Package requirements (dbus-1 >= 1.8) were not met:
  
  No package 'dbus-1' found

zypper in dbus-1-devel
Leap 15.3: dbus-1-devel-1.12.2-150100.8.14.1
Leap 15.4: dbus-1-devel-1.12.2-150400.18.5.1

Also requires glib2-devel

phillxnet · 2022-11-10T19:52:01Z

distro from PyPi

from: https://pypi.org/project/distro/1.6.0/
this version looks to still support Python 2.7 and has Python 3 compatibility also.
And from the canonical Changelog here: https://github.com/python-distro/distro/blob/master/CHANGELOG.md
we have: https://github.com/python-distro/distro/blob/master/CHANGELOG.md#170-20220215

1.7.0 Feb 15, 2022

BACKWARD COMPATIBILITY:

Dropped support for EOL Pythons 2.7, 3.4 and 3.5

N.B. it looks like we were also depending on an old and considered buggy behaviour noted in 1.7.0 changelog:
python-distro/distro#319

Fixed OpenSuse >= 15 support

1.6.0 - Jul 30, 2021

https://github.com/python-distro/distro/blob/master/CHANGELOG.md#160-2021730

Deprecated Python 2.7, 3.4 and 3.5 support. Further releases will only support Python 3.6+

With our Leap 15.3 system package installs as follows:

rleap15-3: # zypper se -s distro | grep "python2-distro\|python3-distro"
i+ | python2-distro           | package    | 1.5.0-3.5.1                          | noarch | Update repository with updates from SUSE Linux Enterprise 15
v  | python2-distro           | package    | 1.5.0-3.3.1                          | noarch | Leap_15_3
   | python2-distroinfo       | package    | 0.3.0-bp153.1.17                     | noarch | Leap_15_3
i+ | python3-distro           | package    | 1.5.0-3.5.1                          | noarch | Update repository with updates from SUSE Linux Enterprise 15
v  | python3-distro           | package    | 1.5.0-3.3.1                          | noarch | Leap_15_3
   | python3-distroinfo       | package    | 0.3.0-bp153.1.17                     | noarch | Leap_15_3

phillxnet · 2022-11-10T20:08:26Z

Six from PyPi

https://pypi.org/project/six/
Official docs: https://six.readthedocs.io/

six 1.16.0

Looks good and is what we were running via 15.3 packages anyway:

rleap15-3:# zypper se -s python2-six
Loading repository data...
Reading installed packages...

S  | Name        | Type    | Version      | Arch   | Repository
---+-------------+---------+--------------+--------+-------------------------------------------------------------
i+ | python2-six | package | 1.14.0-12.1  | noarch | Update repository with updates from SUSE Linux Enterprise 15
v  | python2-six | package | 1.14.0-10.1  | noarch | Update repository with updates from SUSE Linux Enterprise 15
v  | python2-six | package | 1.14.0-3.2.1 | noarch | Leap_15_3

No equivalent found in Leap 15.4 as of yet!
But there is:

rleap15-4:~ # zypper info python3-six | grep "Name\|Repository\|Version"
Repository     : Main Repository
Name           : python3-six
Version        : 1.14.0-12.1

Which is a little confusing. We may just want to build our own here.

phillxnet · 2022-11-10T20:21:12Z

PyZMQ from PyPi

https://pypi.org/project/pyzmq/#history
docs: https://pyzmq.readthedocs.io/en/latest/
Looks to still be Python 2.7 compatible in latest version of 24.0.1: according to general page text.

But the requires tag in PyPi only states 2.7 on 19.0.2 (Jul 31, 2020) at the latest.
https://pypi.org/project/pyzmq/19.0.2/

Also from the homepage changelog: https://pyzmq.readthedocs.io/en/latest/changelog.html#id20

20.0 (Nov 13, 2020)

Drop support for Python < 3.5 (all versions of Python < 3.6 are EOL at time of release)

Require setuptools to build from source

And on a development Leap 15.3 we have:

rleap15-3: # zypper se -s python2-pyzmq
Loading repository data...
Reading installed packages...

S  | Name                | Type    | Version           | Arch   | Repository
---+---------------------+---------+-------------------+--------+-------------------------------------------------------------
i+ | python2-pyzmq       | package | 17.1.2-3.3.1      | x86_64 | Update repository with updates from SUSE Linux Enterprise 15
v  | python2-pyzmq       | package | 17.0.0-1.25       | x86_64 | Leap_15_3
i+ | python2-pyzmq-devel | package | 17.1.2-3.3.1      | x86_64 | Update repository with updates from SUSE Linux Enterprise 15
v  | python2-pyzmq-devel | package | 17.0.0-bp153.1.16 | x86_64 | Leap_15_3
``

Their docs we have that it builds against the system included.

Supported LibZMQ
PyZMQ aims to support all stable (≥ 3.2.2, ≥ 4.0.1 ) versions of libzmq. Building the same pyzmq against various versions of libzmq is supported, but only the functionality of the linked libzmq will be available.

Note

libzmq 3.0-3.1 are not supported, as they never received a stable release.

Binary distributions (wheels on PyPI) of PyZMQ ship with the stable version of libzmq at the time of release, built with default configuration, and include CURVE support provided by tweetnacl. For pyzmq-25.0.0b1, this is 4.3.4.

So we may want to specify a binary wheel variant here.

phillxnet · 2022-11-11T13:25:46Z

Python requests

We already package bundle 2.25.1 (2020-12-16)
As from 2.28.0 (2022-06-09)
https://github.com/psf/requests/blob/main/HISTORY.md

Requests has officially dropped support for Python 2.7. ...

Previous version was 2.27.1 (2022-01-05) so last to support Python 2

Newer versions in Python 3 variant no longer use chardet by default.

Update setup.py and our django-hack.py to accommodate embedding and updating: - pytz to latest version. - six to latest version. - distro to latest version with prior expected behaviour. The following were all moved to the last Python 2 compatible versions, prior to our move to Python 3: - pyzmq - requests - dbus-python

Remove legacy 'distribute' dependency as it's setuptools install clashes/breaks 15.4 build due to conflict with system devel_languages_python setuptools version. Our varying build systems 15.3/15.4 also now produce varying egg paths. Accommodate by overloading.

Now that we are embedding more dependencies we need to pin and add them to our django-hack.py. At least until we replace our build system.

phillxnet · 2022-11-15T18:38:59Z

We are initially sourcing our currently required python2-setuptools via the devel_languages_python OBS repo.

Update setup.py and our django-hack.py to accommodate embedding and updating: - pytz to latest version. - six to latest version. - distro to latest version with prior expected behaviour. The following were all moved to the last Python 2 compatible versions, prior to our move to Python 3: - pyzmq - requests - dbus-python Remove legacy 'distribute' dependency as it's setuptools install clashes/breaks 15.4 build due to conflict with system devel_languages_python setuptools version. Our varying build systems 15.3/15.4 also now produce varying egg paths. Accommodate by overloading. Pin modules required by our embedded Python2 Requests Now that we are embedding more dependencies we need to pin and add them to our django-hack.py. At least until we replace our build system.

…_more_dependencies Embed and update more dependencies #2424

phillxnet · 2022-11-20T16:12:02Z

close by #2431

phillxnet self-assigned this Nov 11, 2022

phillxnet changed the title ~~Move to embed more library dependencies~~ Move to embed and update more library dependencies Nov 11, 2022

phillxnet changed the title ~~Move to embed and update more library dependencies~~ Move to embed and update more dependencies Nov 11, 2022

phillxnet mentioned this issue Nov 11, 2022

update use of python distro module re 1.7.0+ changes #2426

Closed

phillxnet mentioned this issue Nov 11, 2022

Embed and update more dependencies #2424 #2427

Closed

phillxnet added a commit to phillxnet/rockstor-core that referenced this issue Nov 13, 2022

fix typo re idna version rockstor#2424

abc5f13

phillxnet mentioned this issue Nov 15, 2022

Inadvertent sssd related calls in Leap 15.4 #2429

Closed

phillxnet mentioned this issue Nov 18, 2022

Embed and update more dependencies #2424 #2431

Merged

phillxnet added a commit that referenced this issue Nov 19, 2022

Merge pull request #2431 from phillxnet/2424_Move_to_embed_and_update…

81de2c7

…_more_dependencies Embed and update more dependencies #2424

phillxnet closed this as completed Nov 20, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Move to embed and update more dependencies #2424

Move to embed and update more dependencies #2424

phillxnet commented Nov 10, 2022 •

edited

Loading

phillxnet commented Nov 10, 2022 •

edited

Loading

phillxnet commented Nov 10, 2022 •

edited

Loading

phillxnet commented Nov 10, 2022

phillxnet commented Nov 10, 2022 •

edited

Loading

phillxnet commented Nov 11, 2022

phillxnet commented Nov 15, 2022

phillxnet commented Nov 20, 2022

Move to embed and update more dependencies #2424

Move to embed and update more dependencies #2424

Comments

phillxnet commented Nov 10, 2022 • edited Loading

phillxnet commented Nov 10, 2022 • edited Loading

Re: current system package: python-dbus-python

dbus Python Bindings

1.2.18 (2021-07-20)

1.3.0 (2022-09-06)

New dev dependency

phillxnet commented Nov 10, 2022 • edited Loading

distro from PyPi

1.7.0 Feb 15, 2022

1.6.0 - Jul 30, 2021

phillxnet commented Nov 10, 2022

Six from PyPi

six 1.16.0

phillxnet commented Nov 10, 2022 • edited Loading

PyZMQ from PyPi

20.0 (Nov 13, 2020)

phillxnet commented Nov 11, 2022

Python requests

phillxnet commented Nov 15, 2022

phillxnet commented Nov 20, 2022

phillxnet commented Nov 10, 2022 •

edited

Loading

phillxnet commented Nov 10, 2022 •

edited

Loading

phillxnet commented Nov 10, 2022 •

edited

Loading

phillxnet commented Nov 10, 2022 •

edited

Loading