Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RFC: Packages as (optional) namespaces #3243

Merged
merged 36 commits into from
Mar 11, 2024

Conversation

Manishearth
Copy link
Member

@Manishearth Manishearth commented Mar 9, 2022

🖥️ Rendered 🖥️

crates.io FCP
cargo FCP
lang sign off
compiler sign off

Rust Issue: rust-lang/rust#122349


This was previously discussed as a pre-RFC here, and later iterated upon in this repo.

There is a prototype of this RFC available on https://cratespaces.integer32.com/ with instructions here

This RFC brings forth a long-discussed and desired feature: some kind of namespacing for crates.io, achieved via treating "toplevel" packages themselves as namespaces, with ownership being inherited.

Please read before participating

I'm going to include some of the same disclaimers I included in the pre-rfc:

The general topic of namespacing is one that has been discussed many times. In my understanding, a lot of the tension here comes from people having different problems they wish to solve, and these discussions not always acknowledging that, leading to people feeling unheard.

This proposal comes out of many years of one on one discussion, reading threads, and incubation. I do think this is a viable and good path forward, but for this to work out we need to be our best selves here.

I would like to request everyone to keep this discussion constructive and respectful. If a discussion is getting super involved, it may be worth opening a discussion issue on https://github.com/Manishearth/namespacing-rfc/ and linking to it instead.

This RFC is attempting to solve the problem of making crate ownership clear in terms of other crates; it is not attempting to solve the general problem around the squatting of root crate names. As such, if a crate can be published today under a particular name by anyone, it would also also be publishable under the same name by anyone in the world of this RFC proposal. I would prefer if we did not spend much time discussing what I see as the radically different problem of squatting in general.

Thank you all in advance for what I hope will be a constructive and fruitful RFC!

Update: Discussions about separator choice are being conducted here.

@Manishearth Manishearth added the T-crates-io Relevant to the crates.io team, which will review and decide on the RFC. label Mar 9, 2022

A different separator might make more sense.

We could continue to use `/` but also use `@`, i.e. have crates named `@foo/bar`. This is roughly what npm does and it seems to work. The `@` would not show up in source code, but would adequately disambiguate crates and features in Cargo.toml and in URLs.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am rather partial to this approach; it has a bunch of benefits:

  • It has a clear mapping to crates.io/docs.rs URLs
  • There's no ambiguity in feature specs
  • We could use the same syntax directly in source (@ <ident> / triggers parsing as a namespaced path) though this might be too large a syntax change

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do think it would be ideal for the same syntax to be used both in Cargo.toml and in source code. I'd like to suggest the usage of ::: but I think it has the same issue that led to ... becoming ..=.

Can you expand on why @foo::bar might be too large a syntax change?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because it changes what counts as an identifier in Rust: it's a change at the token level, which in turn affects proc macros and such.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(it doesn't have to be at the token level tbh, but that does still complicate how paths work)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would be nice to have @rust-lang/lang input on this as a syntax change

(if this RFC starts going down this route I'll of course add them as an approving team)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what does @ mean in npm's context?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what does @ mean in npm's context?

The way npm does package names in a package.json is like @scope/package@version.
Where scope is a existing organization on npm and @version a published version.

Though they also allow you to use Github repos as shown in the image
image

Copy link

@nbittich nbittich Mar 11, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please don't introduce new keywords. I like the idea of namespace, but please take a simple path, or don't implement it at all until you find one. rust is already complex enough, I'm affraid we have to write things like this in the future:

from crate<'a,T> import T::foo::<'a> where T: Pin<Box<dyn Crate>>> + 'a Namespace + Copy + Debug as @foo

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using a character that is currently not possible within a crate name does avoid issues like a bunch of crates already being published with a name that want as a root. For example, if your org uses something like foo_bar and foo_baz but someone else already published foo, you can't use those names. But you could for something like @foo/bar.

That said, I see @ehuss's comment below that cargo has already closed on support for ::. Does that effectively mean this RFC is decided?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Its been approved by 2 of the 3 related teams and is just waiting on enough votes from the last team.

@steffahn
Copy link
Member

steffahn commented Mar 9, 2022

🤔... semi-serious idea: perhaps backslash could be used in code: foo\bar. That doesn't even tokenize at the moment, so it's fully backwards compatible. Too bad neither URLs not toml supports those.

@jklamer
Copy link

jklamer commented Mar 9, 2022

Thank you for preparing this! I'd like to second the support for ::: as both cargo and path syntax. And maybe even go as far to push back on the idea that it is confusing (or more confusing than other options). "This same thing looks different in cargo.toml than in use statements" is a larger source of confusion for me and my lizard brain than most other things. I may be missing some expertise for the true confusion of :::

@jhpratt
Copy link
Member

jhpratt commented Mar 10, 2022

I don't have time to look through the RFC this moment, but I'm hoping this will be forward compatible with intra-package libraries? I described my thoughts a bit in this issue.

@Manishearth
Copy link
Member Author

@jhpratt i think so

@yigal100
Copy link

yigal100 commented Mar 10, 2022

Cargo and rustc have different roles (bounded contexts) and I find it incredible we keep trying to shoehorn the same design into both.

Cargo must use full paths to namespaced crates for obvious reasons. However, this isn't the right design for use within rust code:

  • A single project only has a tiny subset of the entire crates eco-system. The chances of a name conflict are slim to none.
  • It's highly unlikely a user would need two competing implementations from two orgs within the same project.
  • Adding some sort of separator mapping to code adds redundant complexity (magic) to an already complicated language and would likely introduce edge cases - what if a crate name itself contains an underscore?
  • Duplicating the full crate namespace for each use statement is redundant verbiage. If the user wants to move a crate to another namespace, they'll need to touch many places in the code instead of just a single line in the cargo.toml configuration.

To summarise, cargo should be the sole manager of this concern and this should not bleed into code. I should be able to add a dependency of "foo/bar/baz" to cargo, and expect to use "baz" as the crate name in rust code. In the unlikely event There is a name conflict, I should be able to rename the crate in cargo config - that should be an already existing feature of cargo.

For example, a company might choose to have a naming scheme like:
"Company/product/libfoo"
"Company/product/bar"
And say bar depends on libfoo.

What happens if the company decides to rebrand "product"? Seems to me this is redundant busy-work to go and update all references to libfoo when this could have been a simple config file change.

Final edit:
To preempt possible "OSS" objections to this "comercial" use case as invalid:

  • Enterprise adoption is already happening and is crucial to Rust's success
  • this is as much about the implementation of cargo as it is about crates.io and cargo already supports 3rd party registries in stable.

@Manishearth
Copy link
Member Author

Manishearth commented Mar 10, 2022

@yigal100 I think you have misunderstood the problem people are attempting to resolve by making the syntax the same in Cargo and rustc.

I'm also going to somewhat preemptively ask you to tone it down a bit. Using language like "I find it incredible that..." is feigning surprise and is in general not a constructive way to engage. Consider that you may have misunderstood, or that the people you are discussing with hold different values. If you find this pushback to be harsh, bear in mind that this topic has an extremely hard time being discussed, and even minor unconstructive comments can and will snowball. Either people are on their best behavior or this discussion goes nowhere for the millionth time, and I truly wish for the former.

I am super sympathetic to the goal of separating rustc and Cargo here (indeed, the current RFC draft does not propose any rustc changes, just talks about them in the alternatives: I share this goal to some degree!), however I urge folks to be constructive when talking about it.

To address some points you brought up specifically

What happens if the company decides to rebrand "product"? Seems to me this is redundant busy-work to go and update all references to libfoo when this could have been a simple config file change.

This already exists. You can set the name Cargo passes down to rustc.

  • A single project only has a tiny subset of the entire crates eco-system. The chances of a name conflict are slim to none.
    It's highly unlikely a user would need two competing implementations from two orgs within the same project.

The argument is more about the different syntaxes being confusing (people already have trouble with - and _). At no point is it said that conflicts are the problem (conflicts are resolved with the fact that you can set the name rustc sees in Cargo.toml), you are addressing a straw man here. Conflicts are talked of as a problem only inasmuch as it enables dash typosquatting, where malicious people will take foo-bar and hope that people looking for foo/bar will sometimes make a mistake.

Duplicating the full crate namespace for each use statement is redundant verbiage. If the user wants to move a crate to another namespace, they'll need to touch many places in the code instead of just a single line in the cargo.toml configuration.

This argument already applies to renaming crates in general; and is not a new problem. Furthermore, this RFC doesn't quite propose any form of "moving" crates between namespaces in the first place. Finally, as already mentioned, there are ways to handle crate renames in just Cargo.toml.

@yigal100
Copy link

yigal100 commented Mar 10, 2022

@Manishearth:

What happens if the company decides to rebrand "product"? Seems to me this is redundant busy-work to go and update all references to libfoo when this could have been a simple config file change.

This already exists. You can set the name Cargo passes down to rustc.

We're in agreement here as I've mentioned the same point myself. My comment is really about the default behaviour. Defaults matter a lot for ergonomic reasons and the best default here is to map to the leaf crate name and not pass along the full path under some transformation.

As you noted yourself, the current scheme is already confusing people with '-' vs. '_' and imho we shouldn't add to that an additional transformation.

@Manishearth
Copy link
Member Author

I consider removing the root crate name to also be a transformation that can potentially be confusing.

A thing that I might not have mentioned in the RFC (but I'll edit it in now, it'l llargely have the text I'm putting below) is that there are, broadly speaking, two ways this feature may be used.

One of the ways is for one organization (say, unicode) to release a lot of crates under the namespace as asserting organizational ownership. Think unicode/segmentation, unicode/line-breaking, unicode/script, etc. In such a case, using just the leaf crate name makes a lot of sense.

However, there's another case: where a project wishes to use namespaces to talk about a related set of crates. For example, a very common use case I see is things like serde/derive and icu/provider, where "derive" and "provider" on their own are rather generic and would be prone to clashes (what derive? provider of what?)

It's still a valid design to just pick leaf nodes and ask people to rename, however. I'll mention that in my edit, I feel like I had talked about this at some stage of the discussions but it may not have made it into the RFC.

@Manishearth
Copy link
Member Author

@yigal100 I've added a section. I do somewhat like the leaf approach and it was closer to my original idea but I found in feedback that people who wanted to use it for the "project" use case did not quite enjoy it. But it's now an explicit section where this topic can be discussed further.

@rpjohnst
Copy link

A minor note on the question of the separator:

Using :: there not only aligns well with existing Rust syntax generally, but I think also aligns well with with 2018 edition use path changes. While in the 2015 edition the crate root is the top-level :: namespace, the 2018 edition pushes the current crate down a level into crate:: to make room for other crates to live next to it. Using :: as the separator would take this one step further, and allow that new top level namespace to contain package groups as well as individual crates.

Long term, migrating to a separator that is the same in Rust source code might even reduce some of the confusion around hyphens- crates that use them for namespacing would no longer need any crate name translation from Cargo.

@Absolucy
Copy link

Agreeing with @rpjohnst, my opinion on how to handle use without syntax changes would be something along the lines of this:

  • Check for a namespace first and prioritize it, ie the namespaced serde::derive takes priority over the root namespace serde::derive in the serde crate
  • Reverse this logic and prioritize root namespace first if the use is prefixed by a ::

So,

use serde::derive;   // resolves to namespace "serde", crate "derive"
use ::serde::derive; // resolves to root namespace, crate "serde", module "derive"

Ultimately I see no better way to resolve this without parser/syntax changes, which could require another Rust edition if I'm not mistaken?

@pksunkara
Copy link

pksunkara commented Mar 10, 2022

use serde::derive;   // resolves to namespace "serde", crate "derive"
use ::serde::derive; // resolves to root namespace, crate "serde", module "derive"

There are scenarios where people would want to refer to module "derive" of crate "serde" without the leading ::. Especially in marcos. If you look at clap, we do this exact thing in order to let clap be dependency of another library.

@Manishearth
Copy link
Member Author

Would folks proposing :: be willing to come up with a proposal for :: that includes all the different aspects that need to be addressed? I'd also love to handle it with :: but I'm worried about confusion and ambiguity; and in the past I recall us having a hard time coming up with something that worked with ::.

@Manishearth
Copy link
Member Author

If separators are going to be a major discussion I'm also happy to direct people to Manishearth/namespacing-rfc#1 and/or Manishearth/namespacing-rfc#2 and request in-depth discussion occur there.

@joshtriplett
Copy link
Member

@Manishearth I posted a sketch of a proposal for :: at Manishearth/namespacing-rfc#1 (comment) . Happy to continue there.

@Manishearth
Copy link
Member Author

Yeah, let's consolidate discussion on separator choice there. I've updated the RFC with links to the issue, further comments about separator choice may be hidden and redirected there instead (until we reach consensus there).


If you end up in a situation where you have both `foo/bar` and `foo-bar` as active dependencies of your crate, your code will not compile and you must [rename](https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html#renaming-dependencies-in-cargotoml) one of them.

The `features = ` key in Cargo.toml continues parsing `foo/bar` as "the feature `bar` on dependency `foo`", however it now will unambiguously parse strings ending with a slash (`foo/` and `foo/bar/`) as referring to a dependency, as opposed to feature on a dependency. Cargo may potentially automatically handle the ambiguity or error about it.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another option for disambiguation here could be namespaced features. If namespaced crates did not automatically create an unnamespaced feature like current optional dependencies do, then you would have to use dep:foo/bar to refer to it and foo/bar would unambiguously refer to the feature bar on crate foo.

@tarcieri
Copy link

tarcieri commented Mar 1, 2024

@jtgeibel there was previous discussion regarding Windows paths here: https://github.com/rust-lang/rfcs/pull/3243/files#r983881223

@epage
Copy link
Contributor

epage commented Mar 1, 2024

@jtgeibel the index / filename issues are marked for resolving during stabilization by being in the "Unresolved questions" section. Is that sufficient?

As for dep:foo vs dep::foo, that seems like more of a concern for the cargo team than crates.io for deciding whether that is a blocking concern or not. We ran three separate FCPs for this RFC so we could make sure that all respective teams signed off.

@jtgeibel
Copy link
Member

jtgeibel commented Mar 1, 2024

in the "Unresolved questions" section. Is that sufficient?

Somehow I missed that. Yeah, that is fine with me.

For dep:foo I wasn't intending to raise that as a blocking concern (at least from my perspective). Looks good to me then, so checking my box now.

@rfcbot rfcbot added final-comment-period Will be merged/postponed/closed in ~10 calendar days unless new substational objections are raised. and removed proposed-final-comment-period Currently awaiting signoff of all team members in order to enter the final comment period. labels Mar 1, 2024
@rfcbot
Copy link
Collaborator

rfcbot commented Mar 1, 2024

🔔 This is now entering its final comment period, as per the review above. 🔔

@Manishearth
Copy link
Member Author

Somehow I missed that. Yeah, that is fine with me.

Thanks! Yeah I feel like the nitty gritties of the exact implementation strategy especially for Windows doesn't really need to be RFCd, y'all can pick something when the time comes. I was trying not to overconstrain the teams with this RFC.

@rfcbot rfcbot added finished-final-comment-period The final comment period is finished for this RFC. and removed final-comment-period Will be merged/postponed/closed in ~10 calendar days unless new substational objections are raised. labels Mar 11, 2024
@rfcbot
Copy link
Collaborator

rfcbot commented Mar 11, 2024

The final comment period, with a disposition to merge, as per the review above, is now complete.

As the automated representative of the governance process, I would like to thank the author for their work and everyone else who contributed.

This will be merged soon.

@oli-obk oli-obk merged commit 9d97626 into rust-lang:master Mar 11, 2024
@oli-obk oli-obk deleted the namespacing branch March 11, 2024 22:31
@simonsan
Copy link

🚀 Wow! Thanks for all your work! <3

@@ -0,0 +1,241 @@
- Feature Name: `packages_as_namespaces`
- Start Date: (fill me in with today's date, 2022-03-09)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you forget to change this?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed via #3586

@mattheww
Copy link

I noticed that #3243 (comment)

wanted a note to be added on the tracking issue:

There was some discussion about the exact name conflict rules, and we'd like to see a note in the eventual tracking issue that it's important that those get decided explicitly and written down

which hasn't appeared.

(I would mention it on that issue, but it's locked.)

@Manishearth
Copy link
Member Author

Good catch, done!

@cbiffle
Copy link

cbiffle commented May 7, 2024

I'm not sure if comments like this are helpful, but: I've read through the RFC as merged, and the proposal would almost certainly cover the needs of both operating systems I work on. On Hubris in particular, this is a missing piece we've been trying to figure out before publishing any of the operating system to crates.io (currently it's all git deps or path deps to keep interlopers from sneaking crates in).

So, thank you for threading the needle on this, this solves a real problem in a thoughtful way, and I'm excited to see where this goes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
disposition-merge This RFC is in PFCP or FCP with a disposition to merge it. finished-final-comment-period The final comment period is finished for this RFC. T-crates-io Relevant to the crates.io team, which will review and decide on the RFC. to-announce
Projects
None yet
Development

Successfully merging this pull request may close these issues.