Friday, March 03, 2006

Save Lisp And Die

So yeah, almost an entire year after Noodle was "almost ready to be released", it's.. well, it's almost ready to be released. I haven't had as much chance to work on it as I'd like (and am unlikely to have, as long as I require income), but it has continued to get better and I still quite like working on it and in it (a good sign).

LNoodle vs. PyNoodle

Python plus the benefits of Lisp is great, but the possibility has been nagging at me that Lisp plus the benefits of Python could be even more awesomer. In Python, collections always have sensible iterators that all work the same, indexing and slicing works the same everywhere and is all kinds of easy to write and read, string functionality is hella hella more convenient and simplistic, etc. And our old friend, the Python standard library, is a sign that a benevolent God of computy things smiles down at us. Lisp could use all that, in my opinion, and Python could use the speed and power of a Lisp compiler.

And so-- no, no, wait for it-- I'm implementing Noodle on top of Lisp (sbcl, to be precise). The challenges there are turning out to be very different but still very interesting. There are different forces pulling at the language that way, too; there are things that could be added to Noodle which would be very nice with sbcl underneath, but which might be a pain in the python-based one. I'm not sure yet how much I should indulge in these inviting concessions.

Those of you who have experience with Common Lisp have, I'm sure, heard more harebrained schemes to remake Lisp for the new generation; to bring it into "the present", to make it more accessible or simple or whatever. None of these projects go anywhere, I know. Go ahead and lump this in with the rest of them if you like. I can't offer any reason why Noodle might succeed where others have failed, but it's fun to work on, so I don't care all that much. Plus, it won't do me any good to try and justify Noodle to Lispers-- when they see the dot notation being used to access attributes and (for crying out loud) methods in Lisp, they're all going to hate me. I'm talking despiseage here. But I think I might like having that notation.

Unclear as yet how to proceed with these dual implementations. Maintain them both? More work, but more power. But will it help Noodle grow better if I concentrate on one until it's mature, or will having that portability be an advantage?

Biggest Mistakes So Far

The "trailer" operators I've had in Noodle, indexing and attribute referencing, have got to go. Working them into the Lisp reader would be way, way too much of a pain-- plus I've found there is no order of operations that works very well. One ends up required to write code with full parenthesized forms much too often where prefix syntactic sugar meets postfix.

So, I've come up with some new syntactic sugar for indexing and for attribute access that seems to fill in the gap much more nicely. Here it is:

  • attribute access: .(some expression here).attrname.anothername

    (See that? there's a dot before the attributized expression as well as before each attribute name. Weird, huh? But it seems not to be significantly less convenient than the Python "" and . becomes a prefix operator. Bonus! Another Noodly significant-whitespace rule comes into play here: No whitespace in the middle there, anywhere. "" is not the same as ".foo .bar.baz". The second is two expressions: foo and the baz attribute of bar. A . by itself is a symbol, and names a more flexible but less convenient macro which acts similarly: "(. foo bar baz)".

  • indexing: [(some expression here) 12]

    That is, transform "foo[bar]" from Python the same way we've transformed "foo(bar)". It interferes with the Python-list literal syntax, so that becomes "\[foo bar]", parallel with the tuple literal "\(foo bar)" syntax. Slicing presents just a bit of an issue; should "foo[bar:baz]" become "[foo (: bar baz)]"? That's a good candidate for the "true" parenthesized form, but "[foo (: bar None)]" is mightily more annoying than just "foo[bar:]". Not yet sure if that's worth the non-trivial cost of adding even more syntactic sugar. Another option is "[foo bar baz]", but that doesn't seem to be easily readable as "the slice of foo from bar to baz". Plus, "foo[bar, baz]" is valid Python syntax and should probably be supported, although I haven't seen it used for much.

Compile-time Namespaces?

Macros have posed interesting challenges in terms of namespaces and scope. I think the difficulties have mostly been met. More exploration needs to be done, though. For example, would it be a good idea to allow importing at compile-time- that is, into the compile-time namespace which includes macros? You don't automatically get macro access like "(mymodule.mymacro foo bar)", since the form's first element is an expression, and you'd need to teach the compiler how to recognize when dotted attributes should be retrieved from modules at compile time and when they shouldn't. It seems initially like it would be a mess.

There are also questions about how to reference a shadowed macro: if I define my own macro called case, but I want it to expand eventually to the standard case macro, how do I get that? Do I have to bind it to another name every time I want to do something like that? Maybe that's not too bad. There certainly are times when it would be nice to have symbols bound to packages, though.


Anonymous said...

Long live the noodle!

Piet Delport said...

Paul, have you looked at Scheme's hygienic macro system? They were designed to deal with the same namespace issues mentioned in this post.

paul cannon said...

I don't know much about Scheme's hygienic macros, apparently. I had thought that they were mainly for keeping symbols in a macro definition separate from the surrounding code after expansion. That is, making it so you don't have to (gensym) so much.

If they also solve my problem, then I should certainly look more into them. Thanks for the heads-up.