Import name conflict between two packages

[defnull]

Problem description

I was recently made aware of a name conflict between multipart and python-multipart. Both install themselves as multipart, which causes difficult situations for framework developers now that Python 3.13 removes cgi and the official docs mention multipart as a replacement. I have very good reasons to not just rename multipart into something different and the maintainers for python-multipart also have denied similar requests. There are plans to absorb python-multipart into Starlette (its primary dependent) so the problem might go away on its own, unfortunately without a timeline given.

I'm aware that pypi cannot prevent such conflicts for several (mostly technical) reasons. I'm here to ask for ideas how to solve this in a way that does not make things even worse in the long run.

So, first idea: I could package two files with identical content named multipart.py and multipart_fallback.py into the same wheel and sdist archive and tell users to import the fallback if they run into conflicts. Pro: Easy to maintain on my side. Con: This pollutes the namespace even more and does not feel like the right thing to do.

Second idea: Create a second project named multipart-fallback that depends on multipart and installs a skeleton multipart_fallback.py file with some exec or importlib magic. Pro: Opt-in solution. Con: Harder to maintain, feels like a hack.

Third idea: Do nothing and tell others to vendor multipart.py if they have to. Pro: Not much. Con: A pain for everyone involved.

More ideas?

[henryiii]

The interesting thing is you have multipart.py instead of multipart/__init__.py, which means you don't get an install time error. There will be an error if someone tries to install a file that is not identical over another one. But this case lets you install both. But not sure if there's a way to import multipart.py when there's a multipart/__init__.py, I'm not aware of it.

IMO, multipart owns the PyPI name, so should have the multipart import, and python-multipart should use python_multipart.

[Kludex]

I'll continue to maintain the package even after starlette absorbs it, if there are people using outside the starlette context. At the time I wrote that message, I thought people were not, but it looks like I was wrong, so I'll likely keep maintaining it. Stop mentioning me.

How does multipart became the recommendation on those docs?

[defnull]

But not sure if there's a way to import multipart.py when there's a multipart/__init__.py, I'm not aware of it.

Only by creating a custom Finder and Loader as far as I know. If a module and a package with the same name are installed into the same environment, the package is preferred during import. Not sure if this is part of the import protocol, or just an implementation detail of CPython.

[defnull]

Stop mentioning me.

I'm not mentioning you, I'm referencing public issues for context.

How does multipart became the recommendation on those docs?

I was not involved in that discussion or decision, so I do not know. But it was already mentioned in PEP 594 – Removing dead batteries from the standard library as a possible replacement.

[merwok]

This happened with a bson module when mongodb was fashionable, happened to me recently with dotenv vs django-dotenv, and many other cases.

It is perfectly valid to name a project multipart, pymultipart, multipartlib, python-multipart…
It is perfectly valid for these projects to have a module or package named multipart, pymultipart, multipartlib, python_multipart…

Nowhere in the standard and policies is defined that one of these example projects should own one of these namespaces. Import conflicts without file conflicts are one current problem point of our ecosystem, caused in part by the absence of constraint to match import names to project names.

[sinoroc]

The "easiest" is to rename the project under your control. I assume both project are FLOSS and made by volunteers. So the rules are that you don't owe anything to anyone and no one owes anything to you. That would be my recommendation, remove this headache swiftly. I doubt there is a smart way that will be headache-free.

[henryiii]

It is conventional for a package to share the import name - it's the only way that you can avoid conflicts, since PyPI don't allow conflicts, and it's the only way we currently have to "own" an import name. If one package is abandoned, it can work, though it's still a bit confusing for a user to see import multipart, then run pip install multipart, and get the wrong package! If two packages collide, I would strongly suggest the one that isn't using its PyPI name to handle the situation. Especially since in this case (as an observer who's never used either library or heard of them till today):

• multipart is from 2011, python-multipart is from 2013. So python-multipart was created knowing that multipart was taken and chose to use the conflicting import name they couldn't get on PyPI anyway. Now we have a situation where two libraries can't be installed in the same environment because they chose different dependencies.
• multipart is on version 1.1,0, python-multipart is 0.0.12.

However, on the other hand,

• python-multipart seems more popular, at least historically.

I don't think it's fair for python-multipart to expect multipart to try to work around this issue.

Anyway, some ideas:

multipart could move multipart.py to multipart/__init__.py. That way, you'd get an helpful install-time error instead of a strange import-time error.

Are there overlapping items in both libraries? Either library could build a custom import that combines the items in both libraries if they are both installed - but this would only work if there was no overlap. It would probably have to be a custom loader and .pth file.

python-mulitpart could move to using python_multipart, then provide a back-compat module that is only available if multipart isn't installed. It would probably have to be a custom loader and .pth file.

[mgorny]

To be honest, I feel partially responsible for the situation we're in, and it's quite possible that other Linux distributions are in a similar situation.

FWIU, the rise of popularity of "multipart" packages stem from cgi module deprecation and removal. I've added python-multipart to Gentoo first for the simple reason that starlette depended on it. As a result, when some package used multipart instead, I'd recommend switching over to python-multipart, because we already had it packaged, needed it for starlette and at the time, it seemed better maintained. Which means some people switched to python-multipart for the sake of avoiding the conflict.

Now I am starting feeling like the primary reason that people are using python-multipart is because it was forced on them via the dependency on starlette, and switching over felt like the only way to avoid user-facing conflicts.

That said, given that multipart is now recommended by the documentation and python-multipart seems to be becoming less popular, I'll probably rename python-multipart in Gentoo, and patch starlette locally to aid our users.

[cjwatson]

Regarding how multipart became the recommendation in PEP 594: it's because I put it there, after I'd successfully migrated zope.publisher to it a year and a half previously.

I'm afraid I don't remember whether I even noticed python-multipart at the time; if I did I don't seem to have written it down anywhere, since it didn't come up as one of the practical options available in the zope.publisher discussion. But I generally had a pretty nice experience with multipart so I don't regret the choices I made.

(I'm now a maintainer of multipart, but that dates from after the zope.publisher decision had already been made, as you can see in zopefoundation/zope.publisher#55.)

[defnull]

Update: Thanks to @henryiii we are on a good path towards a solution. The idea is to rename the python-multipart package to python_multipart, but in a backwards compatible way. If only python-multipart is installed, the old import name still works, but prints a helpful warning. This gives dependents enough time to update their import statements and should not break any existing applications. The workaround can be removed after a couple of releases to reach a clean state again. I am really happy with this solution!

[defnull]

I'm really happy how this turned out and think we can close this discussion now. Thanks for all the ideas, feedback, support and contributions. And of cause thanks @Kludex and @henryiii for making this possible.