Namespace package with leading _ in package name is rejected for upload

[aaraney]

Related to:
#14763
#14156

Describe the bug

Cannot upload distribution of namespace package where package name has leading _ (i.e.hydrotools._restclient).

Following the package name normalization specification, the normalized form of hydrotools._restclient is hydrotools-restclient. This means either the packages hydrotools._restclient and hydrotools.restclient are indistinguishable or pypi no longer allows namespace package names with leading _s.

Because of this, I cannot upload a new version of my package.

Expected behavior

Package uploads without error.

To Reproduce

1. Create namespace package with leading _ in package name (not namespace name). For example, namespace._package.
2. python -m build

• build==1.0.3 produces dist/namespace._package-0.0.1-py3-none-any.whl and dist/namespace._package-0.0.1.tar.gz.

3. twine upload dist/* results in:

• HTTPError: 400 Bad Request from https://upload.pypi.org/legacy/
  Start filename for 'namespace.-package' with 'namespace_package'.
  

My Platform

Python 3.9.16

build==1.0.3
twine==4.0.2

Additional context

It seems that PEP 503's normalized names specification differs from package names that were once allowable by PyPI.

[di]

This looks like a bug in pypa/wheel, because https://packaging.python.org/en/latest/specifications/binary-distribution-format/#escaping-and-unicode says:

Each component of the filename is escaped by replacing runs of non-alphanumeric characters with an underscore

Which, based on my interpretation, means it should be producing dist/namespace_package-0.0.1-py3-none-any.whl instead.

[aaraney]

Yeah, I agree with your interpretation @di. It seem then that the specification does not allow for a namespace package to have a leading _ in its name at least in its distribution name. This seems incorrect as it has not been an issue in the past.

[aaraney]

There seems to be disagreement between PEP 508's names section and PEP 503's normalized names section. 508's states:

However, PyPI places strict restrictions on names - they must match a case insensitive regex or they won’t be accepted. Accordingly, in this PEP we limit the acceptable values for identifiers to that regex... The regex (run with re.IGNORECASE) is:

^([A-Z0-9]|[A-Z0-9][A-Z0-9._-]*[A-Z0-9])$

However 503 states:

This PEP references the concept of a “normalized” project name. As per PEP 426 the only valid characters in a name are the ASCII alphabet, ASCII numbers, ., -, and _. The name should be lowercased with all runs of the characters ., -, or _ replaced with a single - character.

This disagreement suggests that python package names are not guaranteed to be unique. Meaning two valid unique python package names that pass the PyPI regex may result in the same package name post normalization.

[aaraney]

Pulling from the list of published packages on PyPI, 28 packages can no longer be updated because of this bug.

0-._.-._.-._.-._.-._.-._.-0
Ahmed-M.-Gamaleldin
AL---Application-Launcher
cmc-csci046-.data-structures
cmc-csci046-.yilinli-trees
EMpy----ElectroMagnetic-Python
E.S.P.-Hadouken
example-pkg.....testing-megankuoo
funniest__test2016
h__ello__worl__d
hydrotools.-restclient
IAF--Interaction-Framework
Jungle-.Py-Compiler
just-a-try--i
jy-.-2019
lh__nester
liuhao__nester
micropython-_ctypes
micropython-_markupbase
nester_-101
pycopy-_ctypes
pyHed---Python-desktop-framework
Quick-.Torrent-Downloader
SPYCE---Python-Server-Pages
TANGO-Project---ALGENCAN
Tzara---A-Personal-Assistant
util--q
Vins__server_messenger

[aaraney]

@di, can you revisit this? In the mean time I have no way to update my package on PyPI.

[di]

This is pypa/wheel#440. Until that issue is resolved, you can work around this by renaming the necessary files yourself:

$ unzip hydrotools._restclient-3.0.5-py3-none-any.whl
$ mv hydrotools._restclient-3.0.5.dist-info/ hydrotools_restclient-3.0.5.dist-info/
$ rm hydrotools._restclient-3.0.5-py3-none-any.whl
$ zip -r hydrotools_restclient-3.0.5-py3-none-any.whl
$ twine upload hydrotools_restclient-3.0.5-py3-none-any.whl

The upload will succeed, the project name will be displayed the same on PyPI and the distribution can be installed as expected (example).

Since PyPI is doing the 'right' thing, per the specification, I don't think we should introduce any changes here to be more lax in what we accept, and we can probably close this in favor of pypa/wheel#440.

[aaraney]

Thanks for the guidance, @di! That solved the issue.