Unable to import Pip-installed & Conda-installed modules in Anaconda managed VirtualEnv environments

[josiah14]

Most of the information you would need should be in the console output, below. I'm running on Ubuntu 18.04 using ZSH as my shell.

hy/sequential/hyid3_and_seq  master ✗                                                                                            18m ⚑
▶ conda activate hy-test
(hy-test)
hy/sequential/hyid3_and_seq  master ✗                                                                                            19m ⚑
▶ pip install pandas
Requirement already satisfied: pandas in /home/josiah/miniconda3/envs/hy-test/lib/python3.7/site-packages (0.25.1)
Requirement already satisfied: pytz>=2017.2 in /home/josiah/miniconda3/envs/hy-test/lib/python3.7/site-packages (from pandas) (2019.2)
Requirement already satisfied: numpy>=1.13.3 in /home/josiah/miniconda3/envs/hy-test/lib/python3.7/site-packages (from pandas) (1.17.2)
Requirement already satisfied: python-dateutil>=2.6.1 in /home/josiah/miniconda3/envs/hy-test/lib/python3.7/site-packages (from pandas) (2.8.0)
Requirement already satisfied: six>=1.5 in /home/josiah/miniconda3/envs/hy-test/lib/python3.7/site-packages (from python-dateutil>=2.6.1->pandas) (1.12.0)
(hy-test)
hy/sequential/hyid3_and_seq  master ✗                                                                                            20m ⚑
▶ which pip
/home/josiah/miniconda3/envs/hy-test/bin/pip
(hy-test)
hy/sequential/hyid3_and_seq  master ✗                                                                                            20m ⚑
▶ hy
hy 0.17.0 using CPython(default) 3.7.4 on Linux
=> (import [pandas :as pd])
Traceback (most recent call last):
  File "stdin-759175d9d1ec0dd5c0ef87379c3ce4cc23835447", line 1, in <module>
    (import [pandas :as pd])
ModuleNotFoundError: No module named 'pandas'
=> (require [pandas :as pd])
Traceback (most recent call last):
  File "stdin-8747230e0bb0ca2c98fa606abb6c56170c498599", line 1, in <module>
    (require [pandas :as pd])
hy.errors.HyRequireError: No module named 'pandas'
=> (exit)
(hy-test)
hy/sequential/hyid3_and_seq  master ✗                                                                                            21m ⚑
▶ python
Python 3.7.4 (default, Aug 13 2019, 20:35:49)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas as pd
>>>

This issue also affects me on 0.16.0, so I'm currently doing my work on version 0.15.0 of Hy so that I can use Pip-installed dependencies from within a Pyenv environment.

[Kodiologist]

Can I ask why you're using Anaconda on Linux? Plain old pip and virtualenv work fine.

[josiah14]

@Kodiologist

It's just what I'm used to working with data scientists. I know the Anaconda syntax for doing things, not as familiar with the raw virtualenv commands for doing the same things.

Should Anaconda matter, though? And why not use Anaconda on Linux? Doesn't it make sense to assume Linux, as any machine I'd want to run a real-world script on would probably run on a, say, for example, AWS VM or Docker Container somewhere that's most likely to be running Linux? I assume you might have thought I would rather pay the Apple Tax for a MacBook or deal with the clunky Microsoft Windows OS when Python bytecode and Anaconda are supposed to support all 3 environments?

I understood Anaconda as generally having a superior installer (ensures compilers like Cython get installed before the packages that require them - I've had issues regarding this on certain libraries with Pip even in the last year that were solved by running conda install instead of pip install to ensure dependencies get installed in the right order) and Anaconda also has a really good set of data science libraries baked in (which is most of what I'm using Python for - data modeling, processing, and analysis). I'm not sure how Linux affects any of these advantages. Another advantage to Anaconda that I'm not sure you get with pyenv is that you can define your Conda environments using a YAML file so that the environment is easily recreatable (good for projects that involve multiple contributors or large teams).

[Kodiologist]

not as familiar with the raw virtualenv commands for doing the same things.

All you have to know is virtualenv $NEW_VIRTUALENV_DIR -p $PYTHON_BINARY, then source ./bin/activate and deactivate.

Should Anaconda matter, though? And why not use Anaconda on Linux?

I don't know. I assume that the reason programming languages have "distributions" (Strawberry Perl, etc.) is because it can be a pain to install things on Windows. I'm a biostatistician and I do fine with pip.

As for what's going on here, I don't know enough about Anaconda to tell, but I assume it has something to do with Anaconda's and Hy's changes to module search not cooperating with each other.

[josiah14]

@Kodiologist In most businesses, once processes are modeled, applications that move to a production environment would almost always get ported to Linux for the cost savings. It might work to assume OSX or Windows in a research environment where the datasets aren't very large or the load on the application isn't very heavy, or you're writing desktop/mobile applications that target a particular platform, but otherwise real-world applications typically end up on Linux eventually, even if initial development is done on Windows or OSX.

Personally, I don't like Microsoft or Apple all that much and have chosen only to run Linux on my personal computers. This also makes sense since I'm currently in the data analytics business and all of the code I've written since 2014 has gotten deployed to Linux environments. Also, my trajectory is to specialize in machine learning and AI applications (which LISP is supposed to be a decent language for), and those applications also would run in Linux environments.

[Kodiologist]

I think you may have misunderstood me. I run Linux exclusively. I was suggesting that language distributions are useful for when you need to get your Linux-developed programs to run on Windows.

[josiah14]

I think language distributions have usefulness beyond getting things to run on Mac and Windows and are more generally useful for setting up development environments and ecosystems under the assumption of a particular usage context. Some of the additional advantages of Anaconda I already mentioned 2 comments ago.

I don't perceive that Anaconda was created only or specifically to get your Linux-developed Python code to run on Windows, but to provide a convenient suite of tools that data scientists can use across platforms, and as an added bonus, it improves upon the Pip installation model which, frankly, has needed some improvements for quite some time.

I made some edits to this comment because the initial language I chose was more heavy-handed than I wanted it to be.

[josiah14]

As for what's going on here, I don't know enough about Anaconda to tell, but I assume it has something to do with Anaconda's and Hy's changes to module search not cooperating with each other.

That's very unfortunate, but I suppose I'll have to live with it. I'll try a raw PyEnv environment and comment back on how that works out for me.

[josiah14]

I found a workaround using pipenv instead of Conda. Macros still don't work in the Jupyter Console (but they haven't for a long time), but they will work in the Jupyter Notebook, so it's not a huge inconvenience.

@Kodiologist Thanks for you quick response and help.

Here's a reference to the order in which things need to be installed in order for Hy to work with Calysto_Hy in Jupyter. Just replace "pip install" with "pipenv install":
Calysto/calysto_hy#15 (comment)

[gabriel-francischini]

I use conda daily just because it had way nicer C/C++ packages support when I needed it (so I've never got familiar with virtualenv even if it did catch up in that aspect, given that conda already fulfills most of my needs).

I've never noticed this issue, so maybe this bug was silently fixed along the way (either by conda, pip, python or hy)?

Here's a conda env where numpy is installed from conda-forge and funcy is installed from pypi:

gabriel@linuxstudio:~$ source activate base
(base) gabriel@linuxstudio:~$ conda activate hy-dev
(hy-dev) gabriel@linuxstudio:~$ python
Python 3.10.4 | packaged by conda-forge | (main, Mar 24 2022, 17:39:04) [GCC 10.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas as pd
>>> from funcy import *
>>> 
(hy-dev) gabriel@linuxstudio:~$ hy
Hy 1.0a4 using CPython(main) 3.10.4 on Linux
=> (import pandas :as pd)
=> (import funcy *)
=> 
now exiting HyREPL...
(hy-dev) gabriel@linuxstudio:~$ conda list
# packages in environment at /home/superwolf/miniconda3/envs/hy-dev:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
ca-certificates           2022.5.18.1          ha878542_0    conda-forge
funcy                     1.17                     pypi_0    pypi
funcy-chain               0.2.0                    pypi_0    pypi
gnureadline               8.1.2                    pypi_0    pypi
hy                        1.0a4                    pypi_0    pypi
hyrule                    0.1                      pypi_0    pypi
ld_impl_linux-64          2.36.1               hea4e1c9_2    conda-forge
libblas                   3.9.0           15_linux64_openblas    conda-forge
libcblas                  3.9.0           15_linux64_openblas    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 12.1.0              h8d9b700_16    conda-forge
libgfortran-ng            12.1.0              h69a702a_16    conda-forge
libgfortran5              12.1.0              hdcd56e2_16    conda-forge
libgomp                   12.1.0              h8d9b700_16    conda-forge
liblapack                 3.9.0           15_linux64_openblas    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libopenblas               0.3.20          pthreads_h78a6416_0    conda-forge
libstdcxx-ng              12.1.0              ha89aaad_16    conda-forge
libuuid                   2.32.1            h7f98852_1000    conda-forge
libzlib                   1.2.12               h166bdaf_0    conda-forge
multipledispatch          0.6.0                    pypi_0    pypi
ncurses                   6.3                  h27087fc_1    conda-forge
numpy                     1.22.4          py310h4ef5377_0    conda-forge
openssl                   3.0.3                h166bdaf_0    conda-forge
pandas                    1.4.2           py310h769672d_2    conda-forge
pip                       22.1.2             pyhd8ed1ab_0    conda-forge
pyrsistent                0.18.1                   pypi_0    pypi
python                    3.10.4          h2660328_0_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python_abi                3.10                    2_cp310    conda-forge
pytz                      2022.1             pyhd8ed1ab_0    conda-forge
readline                  8.1                  h46c0cb4_0    conda-forge
setuptools                62.3.2          py310hff52083_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
sqlite                    3.38.5               h4ff8645_0    conda-forge
tk                        8.6.12               h27826a3_0    conda-forge
toolz                     0.11.2                   pypi_0    pypi
tzdata                    2022a                h191b570_0    conda-forge
wheel                     0.37.1             pyhd8ed1ab_0    conda-forge
xz                        5.2.5                h516909a_1    conda-forge
zlib                      1.2.12               h166bdaf_0    conda-forge
(hy-dev) gabriel@linuxstudio:~$ pip list
Package          Version
---------------- -------
appdirs          1.4.4
colorama         0.4.4
funcparserlib    1.0.0
funcy            1.17
funcy-chain      0.2.0
gnureadline      8.1.2
hy               1.0a4
hyrule           0.1
multipledispatch 0.6.0
numpy            1.22.4
pandas           1.4.2
pip              22.1.2
pyrsistent       0.18.1
python-dateutil  2.8.2
pytz             2022.1
rply             0.7.8
setuptools       62.3.2
six              1.16.0
toolz            0.11.2
wheel            0.37.1
(hy-dev) gabriel@linuxstudio:~$ exit

Is this bug still reproducible to you nowadays, @josiah14 ?
You could try this and see if it fails:

source activate base ; conda activate base
conda create --yes --name replicated-hy python=3.10 pandas
conda activate replicated-hy
pip install --pre hy hyrule funcy
python -c 'import pandas as pd; from funcy import *; print("Python works.")'
hy -c '(import pandas :as pd) (import funcy *) (print "Hy works.")'

[Kodiologist]

I was thinking of checking for myself whether this works now, but Anaconda's terms of use appear to be a somewhat sketchy mutation of a BSD license (which I guess explains why it has no package in the standard Debian and Ubuntu repositories), so I'd rather not have it on my computer. I'm going to take @gabriel-francischini's test for granted. Thanks for doing that.