Improve support for Unicode supplementary characters in identifiers and string interpolation (as in Scala 2)

[som-snytt]

Forward port scala/scala#9805

Fixes #16271

[som-snytt]

[lrytz]

LGTM

[SethTisue]

@som-snytt merge conflict needs resolving

My only other concern before hitting "merge" is to ask whether you looked at whether docs/_docs/internals/syntax.md and/or docs/_docs/reference/syntax.md need to be updated? If you either update those or assure us they don't need updating, then we'll be good to go, here.

[som-snytt]

It looks like the fix on Scala 2 had no impact on the spec because it's "Unicode."

[TheElectronWill]

Could § be allowed too? It seems nonsensical to allow emojis but not the paragraph character, which is on our (ok, on some keyboards).
edit: Ideally, could §someVar be allowed please? 👀 It would make a nice replacement for $ in frontend cases like Laminar.

[som-snytt]

Could § be allowed too? It seems nonsensical to allow emojis but not the paragraph character, which is on our keyboards.

In what context? is it punctuation? Not every character is accepted everywhere.

Arbitrary identifiers:

scala> def `§` = 42
def $u00A7: Int

Also, not on my keyboard.

[som-snytt]

I guess every five days there is a conflicting push to Scanners.

[TheElectronWill]

In what context? is it punctuation? Not every character is accepted everywhere.

Yes, the the section sign is in the "punctuation" category. So are !, ? and /, which are all allowed without backticks :)
For coherence, it could be allowed in (at least) the same places as them.

[raquo]

It would make a nice replacement for $ in frontend cases like Laminar.

To clarify Laminar's use case – we tend to prefix variable and method names with $ as a shorthand for "stream of / signal of". It's an easy way to distinguish between similar but different types at a glance, by name alone, e.g.:

val item: Item = ???
val items: List[Item] = ???
val $item: EventStream[Item] = ???
val $items: EventStream[List[Item]] = ???
val $$items: EventStream[EventStream[List[Item]]] = ???
// etc.

This convention arose because $ is the only symbolic character that alphanumeric identifiers can be prefixed with. It's been very useful that way, until we learned that $ is reserved for compiler internals.

I suggest that it would be useful to allow some "keyboard-typeable" special character to be used as a prefix (or alternatively, as a suffix) to identifiers like variable and method names, and if that's not feasible, to perhaps allow characters like §. Emojis are... cool, but they are too visually disruptive to be used for this purpose in my humble opinion.

We only point to § because it's visually similar to $, I don't know enough about the compiler internals to say what is a good idea for the language, just trying to communicate the problem that the lack of $ presents us.

[som-snytt]

That's an interesting use case. I'd suggest https://github.com/lampepfl/dotty-feature-requests

Alternatively, maybe leading dollar is OK for certain members. A linter could flag violations. That might be an intermediate ground.

[TheElectronWill]

I was thinking about the Laminar case lately, and IMO a good solution would be to use the IDE's capabilities instead of a prefix.
An IDE extension could communicate with the language server to find all the variables of type Stream, Signal, and more, and highlight/underline/whatever-you-want them. It could even display the § prefix, without writing it to the source file 😄

This would eliminate the need for a specific naming convention, and would be customizable to better suit the users' needs.