Namespaced symbols

[khinsen]

This is an idea inspired by #273, but it would solve a wider range of problems.

Python doesn't have namespaced symbols, but that doesn't mean that Hy cannot have them. Let's take some inspiration from Clojure.

There are two kinds of symbols, plain ones and namespaced ones. Namespaced symbols have the syntax namespace/symbol, where namespace would be a full module path in Hy. We could also have a special namespace "hy" for the "built-in" symbols (if, +, etc.) to allow referring to them explicitly.

When the compiler sees a namespaced symbol, it adds the required import to the generated Python code and transforms the slash into a dot. In principle that allows Hy programmers never to use import, but that's not the goal. The main use for namespaced symbols is macros.

Inside a quasiquote, plain symbols are converted to namespaced ones. Each symbol is first looked up in the "hy" namespace, if it isn't there, it is attributed to the namespace in which the macro is defined. Macro writers can use explicitly namespaced symbols if this default doesn't work for them. A plain symbol can be generated inside a quasiquote by quoting it.

As with all automatically generated imports, there is a risk of conflict if a macro expansion in a module generates foo/bar, leading to an implicit (import foo), but the same module also uses foo in some other way, e.g. as a variable. One solution is name mangling for auto-imports, doing something like `import foo as __hy_foo'.

Comments?

[paultag]

I like where this is going. Having it work like you describe avoids the shitty import problem I had earlier. It would also let us move some code out of the compiler (Is this still true?) which is a huge bonus.

One question - what if the module isn't on PYTHONPATH or so? e.g. if we had a Python module imported by filename with importlib provide some macros, what's the fully qualified name? How do we import that in the pyc generated?

I love the direction this is going in.

[khinsen]

As usual the devil is in the details...

My first reaction is to proclaim that using a namespaced symbol implies the same responsibilities as importing a module. So if the module doesn't exist, that's a bug and the just punishment is an ImportError. But I think it's better to look at corner cases once there is a working implementation for the standard ones.

[paultag]

Seems fair enough!

[khinsen]

I have a first working implementation. Everybody is welcome to rip it apart ;-)

[algernon]

While ancient, this'd also be a nice candidate for the Grand Language Cleanup. I'll see if I can pick it up this week.

[Kodiologist]

Inside a quasiquote, plain symbols are converted to namespaced ones. Each symbol is first looked up in the "hy" namespace, if it isn't there, it is attributed to the namespace in which the macro is defined.

Then how would you write an anaphoric macro, or otherwise refer to a symbol in the module where the macro expansion ends up, rather than the module it was defined in?

[Kodiologist]

Anyway, I don't think this is necessary now that we have selective forms of require.

[gilch]

Inside a quasiquote, plain symbols are converted to namespaced ones. Each symbol is first looked up in the "hy" namespace, if it isn't there, it is attributed to the namespace in which the macro is defined.

Then how would you write an anaphoric macro, or otherwise refer to a symbol in the module where the macro expansion ends up, rather than the module it was defined in?

@Kodiologist: they're talking about using Clojure's namespace system, which is quite well thought-out and certainly capable of handling anaphoric macros, even across namespaces. I've done it before. If you emit an unqualified symbol, the current namespace is assumed:

> `x  ; inside syntax quote, so current namespace is applied
clojure.core/x
> `~'x  ; literally emit 'x. Use this for anaphoric macros.
x
> `foo.bar/x  ; you can also explicitly pick a namespace.
foo.bar/x

There's been something of a holy war about lisp-1 vs lisp-2. Clojure's namespace system is the best of both worlds. Hy is just a lisp-1 currently, but Namespaces are one honking great idea -- let's do more of those!

It's not entirely clear how this should interact with Python's module system, but it's far from clear that require is good enough. I think this merits further discussion.

[Kodiologist]

I think what you're describing is orthogonal to the Lisp-1 versus Lisp-2 issue, which has to do with whether function names share the same namespace as ordinary variables.

[gilch]

Yes, that is what Lisp-1/Lisp-2 means. But the Lisp-1/Lisp-2 issue is about the consequences of that design choice, which is not orthogonal at all. Some of the trade-offs are pretty subjective, but I'm talking about the ones that pertain to Clojure's syntax-quote.

To illustrate these consequences, I use some examples from a well-known paper on the subject

 (DEFUN PRINT-SQUARES (LIST)
   (DOLIST (ELEMENT LIST)
     (PRINT (LIST ELEMENT (EXPT ELEMENT 2)))))

This works fine is a Lisp-2. When a symbol is used in the function position it's looked up in the function namespace so there's no collision. But in a Lisp-1, oh noes! You shadowed the LIST function with your LIST parameter. In a Lisp-1 you'd have to do something like this:

(DEFUN PRINT-SQUARES (LST)
   (DOLIST (ELEMENT LST)
     (PRINT (LIST ELEMENT (EXPT ELEMENT 2)))))

We couldn't use the more natural LIST, so we called it LST instead. Modern sytnax highlighting can help you spot when this is necessary. But Clojure can use an explicit namespace instead:

(defn print-squares [list]
  (doseq [element list]
    (println (clojure.core/list element (* element element)))))

It's still not as nice as a Lisp-2, but you do have more opitons than you'd expect from a Lisp-1. I think in this case the difference is a pretty minor issue. A more important advantage of a Lisp-2 is in macros. Lets look at a related example.

Consider the following simple macro.

(DEFMACRO MAKE-FOO (THINGS) `(LIST 'FOO ,THINGS))

Suppose the user of this macro writes (DEFUN FOO (LIST) (MAKE-FOO (CAR LIST))) in a separate file.
You have to know the expansion to see the problem:

* (MACROEXPAND '(MAKE-FOO (CAR LIST)))
(LIST 'FOO (CAR LIST))
T

Again, this works fine in a Lisp-2. The first LIST is looked up in the function namespace, so there's no collision. But in a Lisp-1 like Hy, that's a bug. It can be hard to avoid, since it's hidden behind the macro. I can't call this case minor. This is the main advantage of a Lisp-2, but as noted in the paper, it's not immune to this kind of issue either (e.g. FLET). It just comes up less in practice.

Clojure doesn't have this problem though:

(defmacro make-foo [things] `(list '~'foo ~things))
(defn foo [list] (make-foo (first list)))

user=> (foo '(1 2 3))
(foo 1)
user=> (macroexpand '(make-foo 1))
(clojure.core/list (quote foo) 1)

That demonstrates the real power of Clojure's syntax quote, which automatically inserts the explicit namespaces for you.

One could argue pretty persuasively that Clojure is a Lisp-1. As noted in the paper, however, these are not well defined terms. (Common Lisp, for example, probably has seven namespaces and is flexible enough for the user to define more.) But I would call Clojure a Lisp-n. You get as many namespaces as you want, including for functions. This capability is built in; you don't have to implement it yourself. Clojure has the main advantages of both worlds.

Scheme is a Lisp-1, but can avoid this problem by using its hygenic macro system, which doesn't capture symbols like that. You can't do anaphoric macros that way though.

Hy has none of this. No Clojure-like syntax-quote, just Common Lisp's quasiquote (with clojure-style ~). No Scheme-like hygenic macro system. And no Common Lisp/Lisp-2 function namespace.

Of the three, I like Clojure's solution best, but I'm not sure how Clojure's absolute namespaces should fit into Python's mutable module system. Hy needs a solution.

[Kodiologist]

I see. Well, I would always recommend against using the name of a builtin for something other than a builtin. Using a non-builtin function in the expansion of a macro currently requires including an import in the expansion (unless you're writing an anaphoric macro, of course). It would be nice to not have to do that manually. So I'd guess we'd want to make Hy automatically add the appropriate imports (and convert the call to invoke the module explicitly). Intuitively, this sounds like it would get complicated fast.

[Kodiologist]

Five years later, I still don't think this is happening. Hy is built around the idea that a symbol's name is ultimately just what it looks like. There's probably an alternate universe where this possibility could've been pursued long ago, and Hy would then be different, but at this point it seems very unlikely that something this fundamental will change. I'm not disputing that it could be desirable, though, so people who can do this and make everything work are still welcome to surprise me.