Upgrade dependencies #46

marcofavoritobi · 2023-03-17T14:20:57Z

Proposed changes

Upgrade all dependencies.

GPy issue

Commit fdda8ca makes the project to depend on the GitHub version of GPy. This is needed because of a dependency issue when dealing with the PyPI's version of GPy at version 1.10.0.

The issue arises when we try to add scipy as direct dependency of black-it.

The issue is reproducible on Ubuntu 22.04; run the following commands inside a ubuntu:22.04 container (docker run --rm -it ubuntu:22.04 /bin/bash):

apt update
apt install python3 python3-pip
pip install poetry==1.4.0
cd /root

First, try to create a clean Python environment and install GPy^1.10.0 and latest scipy^1.10.1:

mktmpenv
pip install "GPy~=1.10.0" "scipy~=1.10.1"
# this works

Now, let's try to do the same for a Poetry project.

poetry new gpy-issue
cd gpy-issue
sed -i 's/python = "^3.10"/python = ">=3.10,<3.11"/g' pyproject.toml 
poetry add scipy@^1.10.1
poetry add GPy@^1.10.1

The last command gives the following error:

Because no versions of gpy match >1.10.0,<2.0.0
 and gpy (1.10.0) depends on scipy (>=1.3.0,<1.5.0), gpy (>=1.10.0,<2.0.0) requires scipy (>=1.3.0,<1.5.0).
So, because gpy-issue depends on both scipy (^1.10.1) and gpy (^1.10.0), version solving failed.

Instead, installing GPy from GitHub does not give such error.

Fixes

n/a

Types of changes

What types of changes does your code introduce?
Put an x in the boxes that apply

Bugfix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist

Put an x in the boxes that apply.

I have read the CONTRIBUTING doc
I am making a pull request against the main branch (left side). Also you should start your branch off our main.
Lint and unit tests pass locally with my changes
I have added tests that prove my fix is effective or that my feature works

Further comments

n/a

codecov-commenter · 2023-03-17T14:25:54Z

Codecov Report

Merging #46 (9da4832) into main (3fd8000) will not change coverage.
The diff coverage is 75.00%.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

Additional details and impacted files

@@           Coverage Diff           @@
##             main      #46   +/-   ##
=======================================
  Coverage   96.82%   96.82%           
=======================================
  Files          30       30           
  Lines        1512     1512           
=======================================
  Hits         1464     1464           
  Misses         48       48

Impacted Files	Coverage Δ
black_it/calibrator.py	`97.48% <0.00%> (ø)`
black_it/loss_functions/base.py	`100.00% <ø> (ø)`
black_it/loss_functions/gsl_div.py	`100.00% <ø> (ø)`
black_it/loss_functions/likelihood.py	`85.41% <0.00%> (ø)`
black_it/loss_functions/msm.py	`86.88% <ø> (ø)`
black_it/plot/plot_results.py	`100.00% <ø> (ø)`
black_it/samplers/base.py	`100.00% <ø> (ø)`
black_it/samplers/gaussian_process.py	`98.61% <100.00%> (ø)`
black_it/samplers/xgboost.py	`98.21% <100.00%> (ø)`
black_it/utils/base.py	`100.00% <100.00%> (ø)`
... and 2 more

This is due to a version conflict issue with scipy during poetry's locking. This commit makes the project to depend on the GitHub version of GPy. This is needed because of a dependency issue when dealing with the PyPI's version of GPy at version `1.10.0`. The issue arises when we try to add `scipy` as direct dependency of `black-it`. The issue is reproducible on Ubuntu 22.04; run the following commands inside a `ubuntu:22.04` container (`docker run --rm -it ubuntu:22.04 /bin/bash`): ``` apt update apt install python3 python3-pip pip install poetry==1.4.0 cd /root ``` First, try to create a clean Python environment and install `GPy^1.10.0` and latest `scipy^1.10.1`: ``` mktmpenv pip install "GPy~=1.10.0" "scipy~=1.10.1" ``` Now, let's try to do the same for a Poetry project. ``` poetry new gpy-issue cd gpy-issue sed -i 's/python = "^3.10"/python = ">=3.10,<3.11"/g' pyproject.toml poetry add scipy@^1.10.1 poetry add GPy@^1.10.1 ``` The last command gives the following error: ``` Because no versions of gpy match >1.10.0,<2.0.0 and gpy (1.10.0) depends on scipy (>=1.3.0,<1.5.0), gpy (>=1.10.0,<2.0.0) requires scipy (>=1.3.0,<1.5.0). So, because gpy-issue depends on both scipy (^1.10.1) and gpy (^1.10.0), version solving failed. ``` Instead, installing `GPy` from GitHub does not give such error.

This change was due since in several places the software imports directly from scipy.

To reproduce the changes, use Poetry version specifier "^8.0.4" for ipywidgets: poetry add ipywidgets@^8.0.4

To reproduce the changes, use Poetry version specifier "^3.7.1" for matplotlib: poetry add matplotlib@^3.7.1

upgrade tables from 3.7.0 to 3.8.0 To reproduce the changes, use Poetry version specifier "^3.8.0" for tables: poetry add tables@^3.8.0

To reproduce the changes, use Poetry version specifier "^1.2.2" for scikit-learn: poetry add scikit-learn@^1.2.2

To reproduce the changes, use Poetry version specifier "^1.5.3" for pandas: poetry add pandas@^1.5.3

To reproduce the changes, use Poetry version specifier "^0.12.2" for seaborn: poetry add seaborn@^0.12.2

To reproduce the changes, use Poetry version specifier "^4.4.7" for tox: poetry add --dev tox@^4.4.7

This change was required by the tox version upgrade

To reproduce the changes, use Poetry version specifier "^23.1.0" for black: poetry add --dev tox@^23.1.0

Upgrades: - flake8: 4.0.1 -> 5.0.4 - flake8-bugbear: 22.12.6 -> 23.3.12 - flake8-isort: 4.2.0 -> 6.0.0

Only errors of type B028 happened. From the docs (https://github.com/PyCQA/flake8-bugbear): B028: No explicit stacklevel keyword argument found. The warn method from the warnings module uses a stacklevel of 1 by default. This will only show a stack trace for the line on which the warn method is called. It is therefore recommended to use a stacklevel of 2 or greater to provide more information to the user.

More info: https://pylint.readthedocs.io/en/latest/user_guide/messages/refactor/use-dict-literal.html TL;DR: they shows an 18% increase in performance when using a literal over the constructor in python version 3.10.6.

- markdown-include: 0.6.0 -> 0.8.1 - mkdocs: 1.3.0 -> 1.4.2 - mkdocs-material: 8.2.15 -> 9.1.3 - mkdocs-material-extensions: 1.0.3 -> 1.1.1 - mkdocstrings: 0.18.1 -> 0.20.0 - mkdocstrings-python-legacy: 0.2.2 -> 0.2.3 - mknotebooks: 0.7.1.post0.dev98 -> 0.7.1.post0.dev808 - pymdown-extensions: 9.4 -> 9.10

This was a due change becase we use joblib as a direct dependency.

AldoGl

Great job, a much needed update!

AldoGl · 2023-03-20T11:17:19Z

This I think can be surely merged

marcofavoritobi force-pushed the chore/upgrade-deps branch 9 times, most recently from 5b130c7 to bdb14b5 Compare March 17, 2023 17:12

marcofavoritobi marked this pull request as ready for review March 17, 2023 19:52

marcofavoritobi requested review from AldoGl and muxator March 17, 2023 19:52

This was referenced Mar 17, 2023

Update copyright notice for year 2023 #43

Merged

Add extras mechanism for finer-grained dependency selection #45

Open

marcofavorito added 15 commits March 18, 2023 19:35

chore: add scipy as direct dependency

a23b557

This change was due since in several places the software imports directly from scipy.

upgrade ipywidgets version from 7.7.3 to 8.0.4

ba16825

To reproduce the changes, use Poetry version specifier "^8.0.4" for ipywidgets: poetry add ipywidgets@^8.0.4

upgrade matplotlib from 3.5.2 to 3.7.1

2cd911e

To reproduce the changes, use Poetry version specifier "^3.7.1" for matplotlib: poetry add matplotlib@^3.7.1

upgrade tables from 3.7.0 to 3.8.0

a93ea20

upgrade tables from 3.7.0 to 3.8.0 To reproduce the changes, use Poetry version specifier "^3.8.0" for tables: poetry add tables@^3.8.0

upgrade scikit-learn from 1.1.0 to 1.2.2

7dc550b

To reproduce the changes, use Poetry version specifier "^1.2.2" for scikit-learn: poetry add scikit-learn@^1.2.2

upgrade pandas from 1.4.2 to 1.5.3

457923d

To reproduce the changes, use Poetry version specifier "^1.5.3" for pandas: poetry add pandas@^1.5.3

upgrade seaborn from 0.11.2 to 0.12.2

33c2648

To reproduce the changes, use Poetry version specifier "^0.12.2" for seaborn: poetry add seaborn@^0.12.2

chore: run 'poetry lock' with poetry==1.4.0

bbd2eeb

upgrade tox from 3.28.0 to 4.4.7

56adfd2

To reproduce the changes, use Poetry version specifier "^4.4.7" for tox: poetry add --dev tox@^4.4.7

chore: fix spell_check testenv with allowlist_externals

31be703

This change was required by the tox version upgrade

upgrade black from 22.12.0 to 23.1.0

14d5748

To reproduce the changes, use Poetry version specifier "^23.1.0" for black: poetry add --dev tox@^23.1.0

lint: apply black formatter with its new version 23.1.0

a7d8c36

chore: upgrade flake8 and its plugins

88d6a6e

Upgrades: - flake8: 4.0.1 -> 5.0.4 - flake8-bugbear: 22.12.6 -> 23.3.12 - flake8-isort: 4.2.0 -> 6.0.0

marcofavorito added 12 commits March 18, 2023 19:35

fix: upgrade mypy version to 1.1.1 and fix new errors

28a895a

chore: upgrade hypothesis version in pyproject.toml

908b378

chore: upgrade safety version from 1.10.3 to 2.3.4

0e088b1

chore: remove safety ignore flags

45b1f46

chore: upgrade pylint from 2.15.10 to 2.17.0

63c795d

fix: pylint error use-dict-literal / R1735

b3b7755

More info: https://pylint.readthedocs.io/en/latest/user_guide/messages/refactor/use-dict-literal.html TL;DR: they shows an 18% increase in performance when using a literal over the constructor in python version 3.10.6.

fix 'broad-exception-raised' pylint errors

82c5434

chore: upgrade isort version from 5.10.1 to 5.12.0

3e41c67

chore: add joblib as main dependencies

65eca4c

This was a due change becase we use joblib as a direct dependency.

chore: add keywords in pyproject.toml

73f5970

chore: add missing PyPI classifiers

9da4832

marcofavoritobi force-pushed the chore/upgrade-deps branch from 710a61d to 9da4832 Compare March 18, 2023 18:35

marcofavorito mentioned this pull request Mar 19, 2023

chore: update ipywidgets version in pyproject.toml to ^8.0.0. #14

Closed

7 tasks

AldoGl approved these changes Mar 20, 2023

View reviewed changes

marcofavoritobi merged commit a03363f into main Mar 20, 2023

marcofavoritobi deleted the chore/upgrade-deps branch March 20, 2023 11:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade dependencies #46

Upgrade dependencies #46

marcofavoritobi commented Mar 17, 2023 •

edited

Loading

codecov-commenter commented Mar 17, 2023 •

edited

Loading

AldoGl left a comment

AldoGl commented Mar 20, 2023

Upgrade dependencies #46

Upgrade dependencies #46

Conversation

marcofavoritobi commented Mar 17, 2023 • edited Loading

Proposed changes

GPy issue

Fixes

Types of changes

Checklist

Further comments

codecov-commenter commented Mar 17, 2023 • edited Loading

Codecov Report

AldoGl left a comment

Choose a reason for hiding this comment

AldoGl commented Mar 20, 2023

marcofavoritobi commented Mar 17, 2023 •

edited

Loading

codecov-commenter commented Mar 17, 2023 •

edited

Loading