CLI parses unrelated config files and then crashes

[wimglenn]

• Pip version: 9.0.3
• Python version: 3.6.4
• Operating system: Linux

Local project keeps configuration in a [tool:pytest] section of setup.cfg. Can't use pip from within that directory, because it attempts to parse the file incorrectly (using a ConfigParser with interpolation) and then crashes out.

minimal steps to reproduce:

1. create a clean virtualenv, empty directory
2. create this setup.cfg file in cwd:

[tool:pytest]
log_format = %(name)-18s %(levelname)-8s %(message)s

3. run unrelated pip command, e.g. pip install --upgrade pip.

Exception:
Traceback (most recent call last):
  File "/tmp/blah/venv/lib64/python3.6/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/tmp/blah/venv/lib64/python3.6/site-packages/pip/commands/install.py", line 350, in run
    isolated=options.isolated_mode,
  File "/tmp/blah/venv/lib64/python3.6/site-packages/pip/commands/install.py", line 436, in get_lib_location_guesses
    scheme = distutils_scheme('', *args, **kwargs)
  File "/tmp/blah/venv/lib64/python3.6/site-packages/pip/locations.py", line 141, in distutils_scheme
    d.parse_config_files()
  File "/usr/lib64/python3.6/distutils/dist.py", line 402, in parse_config_files
    val = parser.get(section,opt)
  File "/usr/lib64/python3.6/configparser.py", line 800, in get
    d)
  File "/usr/lib64/python3.6/configparser.py", line 394, in before_get
    self._interpolate_some(parser, option, L, value, section, defaults, 1)
  File "/usr/lib64/python3.6/configparser.py", line 427, in _interpolate_some
    "bad interpolation variable reference %r" % rest)
configparser.InterpolationSyntaxError: bad interpolation variable reference '%(name)-18s %(levelname)-8s %(message)s'

That's a valid pytest config section but ConfigParser is getting confused thinking parts of the logging config are interpolation references. I don't know how best to fix this issue because it's actually in distutils.dist.Distribution where they create a ConfigParser and there is not public API to enable/disable interpolation. However, perhaps pip's locations.py shouldn't be attempting to eagerly parse this file in the first place?

Existing workaround: Changing out of the directory that has setup.cfg before using pip.

[benoit-pierre]

Setuptools behavior is the same (see discussion). But pytest uses py.iniconfig hence the incompatible behavior...

[benoit-pierre]

And py.iniconfig is a completly custom implementation...

[pradyunsg]

I'm pretty sure we can try printing a better error message for this case.

This is precisely why I think #3809 (or something similar) is a good idea -- ini is not a standardized format.

[wimglenn]

Could you clear up a curiousity: why does pip want to read ./setup.cfg in the first place - i.e., what contents in setup.cfg could/should alter the outcome of this command:

pip install --upgrade pip  

Maybe I'm missing something obvious here, but it seems like a setup.cfg file present in current directory should only influence a command such as:

pip install .

[RonnyPfannschmidt]

this shoudl be better documented in pytest - if the canonical configfile isnt used, its up to the user not to break the configfiles of other tools

[wimglenn]

Which is canonical? pytest.ini presumably?

[RonnyPfannschmidt]

@wimglenn correct, BTW im one of the pytest maintainers

[wimglenn]

I like having pytest config as a section in setup.cfg because it avoids yet another random config file lying around in the project root (I already need it for setuptools). I see your point but it is also up to other tools not to break for silly reasons .. :) distutils should not try to interpolate, imo, but since it's stdlib it's probably impossible to change now - pip probably shouldn't need to eagerly parse everybody else's sections!

[RonnyPfannschmidt]

@wimglenn until there is a common configformat with a common api, is most sensible to keep things in different files since different things can easily breka each other

hopefully pyproject.toml will sort that out

[gaborbernat]

I've run into this myself. This makes essentially not possible to put Unicode characters inside setup.cfg, for example, the name of the authors if you use configuration based setuptools... 😢 setuptools changed that setup.cfg is always UTF-8... distutils/pip does not make this assumption yet.

[jnoortheen]

I stumbled into the same even in versions 20.*

[uranusjr]

@gaborbernat How did Setuptools communicate the encoding change when it happened? IMO it is reasonable to change to parser encoding to UTF-8, but I don’t want to break whatever is relying on platform encoding.

[gaborbernat]

As far as I remember later setuptools switched to utf 8 encoding so this should not be a problem now with latest pip and setuptools 🤔

[uranusjr]

Hmm, taking another look, this config file parsing logic is buried deep inside distutils. Does Setuptools use its own implementation? Not sure what’s the best course for pip to deal with this…

https://github.com/python/cpython/blob/6000087fe979705dc588a46a35da884cc0198c67/Lib/distutils/dist.py#L402-L419

[gaborbernat]

setuptools has it's own copy of dist; https://github.com/pypa/setuptools/blob/master/setuptools/dist.py#L553; was this release https://setuptools.readthedocs.io/en/latest/history.html#v41-0-0 that made it UTF-8

[uranusjr]

Thanks for the clarification! pip is really short of options here (it can’t even just ignore the offending file, since the loop itself is buried in distutils). I guess it’s not very realistic to have another copy of dist in pip, so we can either push a new encoding argument to stdlib distutils (is this even possible?), or monkey-patch the open function in the module. Neither sounds particularly good 😞

[aucampia]

rather annoying :/

[pradyunsg]

This is no longer an issue in 3.10+. Closing this out to reflect that.

[wimglenn]

Related: #11298