Investigate users with slow import time, and attempt to mitigate in future releases

[kennethreitz]

Letter from a fan:

Hi Kenneth!
Lol I'm named Kenneth too :) - but just call me Ken.
Thanks for Requests! I use it for my bots, and it really simplifies everything.
It helps my life - I work with a MediaWiki wiki, and Requests condenses all the (guess what) > requests I have to make.
My only qualm is how long it takes to import - but I love it all the same.
Thank you!

This pretty much sums up many people's perception of Requests, and I'd like for us to fix this — be it pyopenssl's fault or not.

[methane]

See https://gist.github.com/methane/96d58a29e57e5be97769897462ee1c7e

[sigmavirus24]

@methane can you share how you generated that?

[methane]

This is produced by patched Python.
See python/cpython#3490

[NielsDa]

Hi
Are there any plans to improve this issue?.
request is used in a CLI tool named iocage that is used to control virtual instances of FreeBSD,
Importing requests is significantly slower than creating, and starting a new VM by cloning 100000 files. :-)
Since iocage is started every time a command is issued, it is like a 3-4 sec. delay is introduced into bash, ..pretty annoying..
and in most cases it actually does not need the functionality provided by requests.. :-|
https://forums.freenas.org/index.php?threads/iocage-is-slow-at-loading-request.63109/

[methane]

Is pyOpenSSL installed?
If so, could you try uninstall it?

[NielsDa]

>Is pyOpenSSL installed?
Seems like it.

/usr/local/lib/python3.6/site-packages/pyOpenSSL-16.2.0-py3.6.egg-info
/usr/local/lib/python3.6/site-packages/pyOpenSSL-16.2.0-py3.6.egg-info/PKG-INFO
/usr/local/lib/python3.6/site-packages/pyOpenSSL-16.2.0-py3.6.egg-info/SOURCES.txt
/usr/local/lib/python3.6/site-packages/pyOpenSSL-16.2.0-py3.6.egg-info/top_level.txt
/usr/local/lib/python3.6/site-packages/pyOpenSSL-16.2.0-py3.6.egg-info/requires.txt
/usr/local/lib/python3.6/site-packages/pyOpenSSL-16.2.0-py3.6.egg-info/dependency_links.txt

If so, could you try uninstall it?
I don't think this is allowed on FreeNAS, the host is pretty much hard coded.

[methane]

OK. Maybe, this part causes slowdown.

https://github.com/requests/requests/blob/b66908e7b647689793e299edc111bf9910e93ad3/requests/__init__.py#L93-L102

[methane]

@NielsDa Have you checked #4332 ?
What is version of your pyOpenSSL and cryptography?

[methane]

@NielsDa And would you try #4591 too?

[sigmavirus24]

@NielsDa historically speaking pyOpenSSL and cryptography account for the performance issues you're seeing. The "fix" is to uninstall them. If you can't do so, there's little that this project can do to help you if those turn out to be the culprits

[NielsDa]

What is version of your pyOpenSSL and cryptography?
pyOpenSSL 16.2
cryptography 1.7.2

The "fix" is to uninstall them (pyOpenSSL and cryptography)
Hmm..
FreeNAS come as an appliance, with a fixed set of packages installed, and the packet manager is kind of disabled.
The system is headless with a http GUI written in django/ python.
The root file system is a tempfs (ramdisk) and the contents of /etc is generated from a configuration database on boot.
The functionality provided by request might be used to update the system itself.

[sigmavirus24]

@NielsDa then it seems there's an issue with the configuration of FreeNAS and that should be brought to the attention of that project's maintainers.

[lgommans]

To quantify just how slow it is (it hasn't been mentioned in this thread yet):

$ time python -c 'import requests'
real   0m0.135s
user   0m0.116s
sys    0m0.020s

$ time python3 -c 'import requests'
real   0m0.405s
user   0m0.381s
sys    0m0.025s

Four hundred milliseconds. That's a good reason to find a different library to use (or write it from scratch -- all I need is some GET requests).

For comparison, time python(3) -c 'import sys' takes 29ms and 9ms in python 3 and 2, respectively.

Tested on Debian Buster with requests version:

$ dpkg -l | grep -- -requests
ii  python-requests        2.18.4-2 [...]
ii  python3-requests       2.18.4-2 [...]

My version of the package python-openssl (I'm guessing that's pyOpenSSL) is 17.5.0-1 for both python 2 and 3. Package python-cryptography is 2.1.4-1, again for both python 2 and 3.

[methane]

I tested with clean virtualenv:

$ time py27/bin/python -c 'import requests'

real    0m0.137s
user    0m0.118s
sys     0m0.020s

$ time py37/bin/python -c 'import requests'

real    0m0.183s
user    0m0.172s
sys     0m0.012s

[methane]

And on Ubuntu 17.10:

$ time /usr/bin/python3 -c 'import requests'

real    0m0.145s
user    0m0.115s
sys     0m0.031s

I suspect that something wrong in your environment.

[lgommans]

I'd say >100ms is still ridiculously slow and a good reason to find another library. It's not an order of magnitude off from my 400ms in py3.

(My laptop is otherwise very fast with a good CPU and SSD. Something might be wrong with my setup (Debian Buster is currently in "testing" after all), but I can at least tell that it's not the hardware. And at any rate, 100ms is still ridiculous.)

[methane]

It's difficult to reduce the 100ms. About 60ms of the 100ms is from stdlib's http module.
And 10ms is from chardet. 6ms from cgi...

(After removed pyOpenSSL)
$ time py37/bin/python -c 'import requests'

real    0m0.115s
user    0m0.112s
sys     0m0.004s

$ time py37/bin/python -c 'import http.client, http.cookies, http.cookiejar, cgi, chardet'

real    0m0.081s
user    0m0.065s
sys     0m0.016s

[sigmavirus24]

@lgommans I appreciate that you feel that it's unacceptable. Repeating that sentiment, however, simply won't magically make it faster. If you have something constructive to contribute, by all means continue. If you're going to keep repeating yourself, however, I'd like to point out that there are 1,274 people all getting notifications (of some sort, some including email) about your opinions on the matter. I'm going to take a guess and say that they now have a firm understanding of your position.

[lgommans]

@sigmavirus24 The new information I was trying to add was an actual number, which wasn't posted before in this thread. Doing this in fact revealed that my case is exceptionally slow, so if it turns out not to be an isolated issue, you now have someone to ask in case you want to have someone run tests on such a system.

I appreciate that you feel this was not worth reading for over a thousand people. There aren't a thousand comments so I didn't think to check, but there are actually 4 other participants mentioned on top. Even had I thought of it, I could not have known that >1000 people receive a CC of this. This concept is new to me (whatever concept this is, anyway). Sorry for that. I guess I'll refrain from trying to participate in this project any further because I might be inadvertently talking to thousands of hidden people at the same time, who will become annoyed at getting a natural conversation copied over to their inbox and will post snarky remarks rather than reaching out privately (as to not spam those people further) and letting me know (not in front of the whole group) that there is something I am unaware of. I'm going to take a guess and say you're not aware of how your comment might make people (who come across this thread, let alone to whom it was directed) think twice about adding something that they think is valuable but might, perhaps, not be worth reading for over a thousand people which may or may not be getting it in their inbox.

[sethmlarson]

We've removed the mandatory requirement on two of the heaviest libraries in our dependency tree (pyOpenSSL and idna). If import time is still not up to standards we should reopen a new issue investigating the latest performance.