Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Parsing slow because of noarch build #30

Closed
1 task done
sterlinm opened this issue Feb 18, 2022 · 6 comments · Fixed by #31
Closed
1 task done

Parsing slow because of noarch build #30

sterlinm opened this issue Feb 18, 2022 · 6 comments · Fixed by #31

Comments

@sterlinm
Copy link

sterlinm commented Feb 18, 2022 •
edited
Loading

Issue:

It looks like this recipe only builds the pure python version of pendulum, so the parsing is significantly slower than the version you get from pip which includes the compiled extensions.

String parsing using pendulum installed with conda:

In [1]: from pendulum.parsing import parse_iso8601

In [2]: %timeit parse_iso8601('2021-12-31 12:15:33+00:00')
24.5 µs ± 2.06 µs per loop (mean ± std. dev. of 7 runs, 10,000 loops each)

String parsing using pendulum installed with pip

In [1]: from pendulum.parsing import parse_iso8601

In [2]: %timeit parse_iso8601('2021-12-31 12:15:33+00:00')
292 ns ± 14.9 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)

Here's the difference in the site-packages directory:

(pip_pendulum) jovyan@jupyter-jupyterlab-2djupyterlab-2ddemo-2dvf19hhtr:~$ ls /srv/conda/envs/pip_pendulum/lib/python3.10/site-packages/pendulum/parsing
exceptions  __init__.py  _iso8601.c  _iso8601.cpython-310-x86_64-linux-gnu.so  iso8601.py  __pycache__
(pip_pendulum) jovyan@jupyter-jupyterlab-2djupyterlab-2ddemo-2dvf19hhtr:~$ ls /srv/conda/envs/conda_pendulum/lib/python3.10/site-packages/pendulum/parsing/
exceptions  __init__.py  _iso8601.c  iso8601.py  __pycache__

I haven't created a conda recipe before, but I'm happy to have a go at creating a pull request. The situation is pretty much the same on Windows.


conda_pendulum environment (conda list): 
$ conda list
# packages in environment at /srv/conda/envs/conda_pendulum:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       1_gnu    conda-forge
asttokens                 2.0.5              pyhd8ed1ab_0    conda-forge
backcall                  0.2.0              pyh9f0ad1d_0    conda-forge
backports                 1.0                        py_2    conda-forge
backports.functools_lru_cache 1.6.4              pyhd8ed1ab_0    conda-forge
black                     22.1.0             pyhd8ed1ab_0    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
ca-certificates           2021.10.8            ha878542_0    conda-forge
click                     8.0.3                    pypi_0    pypi
dataclasses               0.8                pyhc8e2a94_3    conda-forge
decorator                 5.1.1              pyhd8ed1ab_0    conda-forge
executing                 0.8.2              pyhd8ed1ab_0    conda-forge
ipython                   8.0.1                    pypi_0    pypi
jedi                      0.18.1                   pypi_0    pypi
ld_impl_linux-64          2.36.1               hea4e1c9_2    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 11.2.0              h1d223b6_12    conda-forge
libgomp                   11.2.0              h1d223b6_12    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libuuid                   2.32.1            h7f98852_1000    conda-forge
libzlib                   1.2.11            h36c2ea0_1013    conda-forge
matplotlib-inline         0.1.3              pyhd8ed1ab_0    conda-forge
mypy-extensions           0.4.3                    pypi_0    pypi
mypy_extensions           0.4.3           py310hff52083_4    conda-forge
ncurses                   6.3                  h9c3ff4c_0    conda-forge
openssl                   3.0.0                h7f98852_2    conda-forge
parso                     0.8.3              pyhd8ed1ab_0    conda-forge
pathspec                  0.9.0              pyhd8ed1ab_0    conda-forge
pendulum                  2.1.2              pyhd8ed1ab_1    conda-forge
pexpect                   4.8.0              pyh9f0ad1d_2    conda-forge
pickleshare               0.7.5                   py_1003    conda-forge
pip                       22.0.3             pyhd8ed1ab_0    conda-forge
platformdirs              2.5.0              pyhd8ed1ab_0    conda-forge
prompt-toolkit            3.0.27             pyha770c72_0    conda-forge
ptyprocess                0.7.0              pyhd3deb0d_0    conda-forge
pure_eval                 0.2.2              pyhd8ed1ab_0    conda-forge
pygments                  2.11.2             pyhd8ed1ab_0    conda-forge
python                    3.10.2          hc74c709_3_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python_abi                3.10                    2_cp310    conda-forge
pytzdata                  2020.1             pyh9f0ad1d_0    conda-forge
readline                  8.1                  h46c0cb4_0    conda-forge
setuptools                60.9.2                   pypi_0    pypi
six                       1.16.0             pyh6c4a22f_0    conda-forge
sqlite                    3.37.0               h9cd32fc_0    conda-forge
stack_data                0.2.0              pyhd8ed1ab_0    conda-forge
tk                        8.6.12               h27826a3_0    conda-forge
tomli                     2.0.1              pyhd8ed1ab_0    conda-forge
traitlets                 5.1.1              pyhd8ed1ab_0    conda-forge
typed-ast                 1.5.2                    pypi_0    pypi
typing_extensions         4.1.1              pyha770c72_0    conda-forge
tzdata                    2021e                he74cb21_0    conda-forge
wcwidth                   0.2.5              pyh9f0ad1d_2    conda-forge
wheel                     0.37.1             pyhd8ed1ab_0    conda-forge
xz                        5.2.5                h516909a_1    conda-forge
zlib                      1.2.11            h36c2ea0_1013    conda-forge

pip_pendulum environment (conda list): 
$ conda list
# packages in environment at /srv/conda/envs/pip_pendulum:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       1_gnu    conda-forge
asttokens                 2.0.5              pyhd8ed1ab_0    conda-forge
backcall                  0.2.0              pyh9f0ad1d_0    conda-forge
backports                 1.0                        py_2    conda-forge
backports.functools_lru_cache 1.6.4              pyhd8ed1ab_0    conda-forge
black                     22.1.0             pyhd8ed1ab_0    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
ca-certificates           2021.10.8            ha878542_0    conda-forge
click                     8.0.3                    pypi_0    pypi
dataclasses               0.8                pyhc8e2a94_3    conda-forge
decorator                 5.1.1              pyhd8ed1ab_0    conda-forge
executing                 0.8.2              pyhd8ed1ab_0    conda-forge
ipython                   8.0.1                    pypi_0    pypi
jedi                      0.18.1                   pypi_0    pypi
ld_impl_linux-64          2.36.1               hea4e1c9_2    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 11.2.0              h1d223b6_12    conda-forge
libgomp                   11.2.0              h1d223b6_12    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libuuid                   2.32.1            h7f98852_1000    conda-forge
libzlib                   1.2.11            h36c2ea0_1013    conda-forge
matplotlib-inline         0.1.3              pyhd8ed1ab_0    conda-forge
mypy-extensions           0.4.3                    pypi_0    pypi
mypy_extensions           0.4.3           py310hff52083_4    conda-forge
ncurses                   6.3                  h9c3ff4c_0    conda-forge
openssl                   3.0.0                h7f98852_2    conda-forge
parso                     0.8.3              pyhd8ed1ab_0    conda-forge
pathspec                  0.9.0              pyhd8ed1ab_0    conda-forge
pendulum                  2.1.2                    pypi_0    pypi
pexpect                   4.8.0              pyh9f0ad1d_2    conda-forge
pickleshare               0.7.5                   py_1003    conda-forge
pip                       22.0.3             pyhd8ed1ab_0    conda-forge
platformdirs              2.5.0              pyhd8ed1ab_0    conda-forge
prompt-toolkit            3.0.27             pyha770c72_0    conda-forge
ptyprocess                0.7.0              pyhd3deb0d_0    conda-forge
pure_eval                 0.2.2              pyhd8ed1ab_0    conda-forge
pygments                  2.11.2             pyhd8ed1ab_0    conda-forge
python                    3.10.2          hc74c709_3_cpython    conda-forge
python-dateutil           2.8.2                    pypi_0    pypi
python_abi                3.10                    2_cp310    conda-forge
pytzdata                  2020.1                   pypi_0    pypi
readline                  8.1                  h46c0cb4_0    conda-forge
setuptools                60.9.2                   pypi_0    pypi
six                       1.16.0             pyh6c4a22f_0    conda-forge
sqlite                    3.37.0               h9cd32fc_0    conda-forge
stack_data                0.2.0              pyhd8ed1ab_0    conda-forge
tk                        8.6.12               h27826a3_0    conda-forge
tomli                     2.0.1              pyhd8ed1ab_0    conda-forge
traitlets                 5.1.1              pyhd8ed1ab_0    conda-forge
typed-ast                 1.5.2                    pypi_0    pypi
typing_extensions         4.1.1              pyha770c72_0    conda-forge
tzdata                    2021e                he74cb21_0    conda-forge
wcwidth                   0.2.5              pyh9f0ad1d_2    conda-forge
wheel                     0.37.1             pyhd8ed1ab_0    conda-forge
xz                        5.2.5                h516909a_1    conda-forge
zlib                      1.2.11            h36c2ea0_1013    conda-forge

Details about conda and system ( conda info ):

$ conda info

     active environment : pip_pendulum
    active env location : /srv/conda/envs/pip_pendulum
            shell level : 3
       user config file : /home/jovyan/.condarc
 populated config files : /srv/conda/.condarc
          conda version : 4.9.2
    conda-build version : not installed
         python version : 3.8.6.final.0
       virtual packages : __glibc=2.27=0
                          __unix=0=0
                          __archspec=1=x86_64
       base environment : /srv/conda  (writable)
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /srv/conda/pkgs
                          /home/jovyan/.conda/pkgs
       envs directories : /srv/conda/envs
                          /home/jovyan/.conda/envs
               platform : linux-64
             user-agent : conda/4.9.2 requests/2.25.0 CPython/3.8.6 Linux/5.4.0-1035-azure ubuntu/18.04.6 glibc/2.27
                UID:GID : 1000:1000
             netrc file : None
           offline mode : False
@xylar
Copy link
Contributor

xylar commented Feb 18, 2022

@sterlinm

I think we would welcome a pull request to build the optimized version of the package, even if it's a starting point. We can tweak or make suggestions as needed. Thank you for bringing this to our attention and volunteering to give a pull request a go.

@sterlinm
Copy link
Author

Great, I'll give it a shot. Thanks!

@ocefpaf
Copy link
Member

ocefpaf commented Feb 18, 2022

Looks like they control it with an env var PENDULUM_EXTENSIONS. We meed to set that up and add the compilers to host.

@sterlinm
Copy link
Author

@xylar or @ocefpaf, are either of you aware of another feedstock that's a good example of translating a poetry package into a conda recipe?

Looking at the pendulum repo, I see that there's a make file that runs the build-wheels.sh script for manylinux_x86_64 and manylinux1_i686. I'm not quite clear on how it's dealing with windows.

Thanks for your help!

@ocefpaf ocefpaf mentioned this issue Feb 18, 2022
Merged
@ocefpaf
Copy link
Member

ocefpaf commented Feb 18, 2022

@xylar or @ocefpaf, are either of you aware of another feedstock that's a good example of translating a poetry package into a conda recipe?

In terms of conda-forge all you need is to add poetry as a build/host dependency. That is already in the recipe.

Looking at the pendulum repo, I see that there's a make file that runs the build-wheels.sh script for manylinux_x86_64 and manylinux1_i686. I'm not quite clear on how it's dealing with windows.

We don't need all the missing part was to set the environment variable that triggers the extension build and, as you reported above, remove the noarch. We also need to add compilers to be able to build the extension. See #31

@sterlinm
Copy link
Author

Thanks very much! I'm sorry I wasn't able to sort out the PR for you but I appreciate your patience :)

@xylar xylar closed this as completed in #31 Feb 19, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Compile extension ocefpaf/pendulum-feedstock
3 participants
@ocefpaf @sterlinm @xylar