Wednesday, October 29, 2008

Closure

Bwah ha ha, see what I did there in the topic? It's a programming-language-nerd pun.

This blog thing deserves some followup. I have abandoned the Noodle effort. It was a terrifically fun project, and I learned a lot about Python internals and about languages in general that I'm not sure I would have been able to learn otherwise. It's just not something that I see going anywhere useful. There are so many exciting and/or interesting developments going on right now with LLVM gaining some traction, PyPy getting momentum, the resurrection of Parrot ("'e's not dead, 'e's pining for the fjords"), Pythonistas and Rubyists actually learning from each other, and so on, that I'm sure any of them have a better chance for making programming life better than Noodle.

S-expressions are still awesome.

Friday, May 12, 2006

Release

I gave a presentation on the work I've done on Noodle at last night's Utah Python User's Group meeting. I made an alpha release of Noodle available to complement that presentation. You can find both a tarball and the slides (OpenOffice format) at xillion.org. The slides are in PDF format on UPyUG's site here.

The conclusion I made in the presentation was that designing a language is far harder than implementing it. I think Noodle works very well -- that is, there is a decent implementation of the language as designed -- but the design is not good enough. The .foo.bar and [foo bar] attribute reference and subscripting syntaxes I mentioned previously have yet to feel comfortable to me. The semantics of macro namespaces leave some cases which are difficult to handle or just plain annoying. There are more warts than those. Noodle is not yet a language I would use in preference to Python or Lisp for general work.

I do still think it could reach that point, but who knows what could change by then? As I mention on the last slide, I reserve the right to change everything quite completely. This release of Noodle is more for the academic interest of others, rather than being a good tool for real work.

Next step is to put up some prettified docs and a tutorial.

Friday, March 03, 2006

Save Lisp And Die

So yeah, almost an entire year after Noodle was "almost ready to be released", it's.. well, it's almost ready to be released. I haven't had as much chance to work on it as I'd like (and am unlikely to have, as long as I require income), but it has continued to get better and I still quite like working on it and in it (a good sign).

LNoodle vs. PyNoodle

Python plus the benefits of Lisp is great, but the possibility has been nagging at me that Lisp plus the benefits of Python could be even more awesomer. In Python, collections always have sensible iterators that all work the same, indexing and slicing works the same everywhere and is all kinds of easy to write and read, string functionality is hella hella more convenient and simplistic, etc. And our old friend, the Python standard library, is a sign that a benevolent God of computy things smiles down at us. Lisp could use all that, in my opinion, and Python could use the speed and power of a Lisp compiler.

And so-- no, no, wait for it-- I'm implementing Noodle on top of Lisp (sbcl, to be precise). The challenges there are turning out to be very different but still very interesting. There are different forces pulling at the language that way, too; there are things that could be added to Noodle which would be very nice with sbcl underneath, but which might be a pain in the python-based one. I'm not sure yet how much I should indulge in these inviting concessions.

Those of you who have experience with Common Lisp have, I'm sure, heard more harebrained schemes to remake Lisp for the new generation; to bring it into "the present", to make it more accessible or simple or whatever. None of these projects go anywhere, I know. Go ahead and lump this in with the rest of them if you like. I can't offer any reason why Noodle might succeed where others have failed, but it's fun to work on, so I don't care all that much. Plus, it won't do me any good to try and justify Noodle to Lispers-- when they see the dot notation being used to access attributes and (for crying out loud) methods in Lisp, they're all going to hate me. I'm talking despiseage here. But I think I might like having that notation.

Unclear as yet how to proceed with these dual implementations. Maintain them both? More work, but more power. But will it help Noodle grow better if I concentrate on one until it's mature, or will having that portability be an advantage?

Biggest Mistakes So Far

The "trailer" operators I've had in Noodle, indexing and attribute referencing, have got to go. Working them into the Lisp reader would be way, way too much of a pain-- plus I've found there is no order of operations that works very well. One ends up required to write code with full parenthesized forms much too often where prefix syntactic sugar meets postfix.

So, I've come up with some new syntactic sugar for indexing and for attribute access that seems to fill in the gap much more nicely. Here it is:


  • attribute access: .(some expression here).attrname.anothername

    (See that? there's a dot before the attributized expression as well as before each attribute name. Weird, huh? But it seems not to be significantly less convenient than the Python "foo.bar.baz" and . becomes a prefix operator. Bonus! Another Noodly significant-whitespace rule comes into play here: No whitespace in the middle there, anywhere. ".foo.bar.baz" is not the same as ".foo .bar.baz". The second is two expressions: foo and the baz attribute of bar. A . by itself is a symbol, and names a more flexible but less convenient macro which acts similarly: "(. foo bar baz)".


  • indexing: [(some expression here) 12]

    That is, transform "foo[bar]" from Python the same way we've transformed "foo(bar)". It interferes with the Python-list literal syntax, so that becomes "\[foo bar]", parallel with the tuple literal "\(foo bar)" syntax. Slicing presents just a bit of an issue; should "foo[bar:baz]" become "[foo (: bar baz)]"? That's a good candidate for the "true" parenthesized form, but "[foo (: bar None)]" is mightily more annoying than just "foo[bar:]". Not yet sure if that's worth the non-trivial cost of adding even more syntactic sugar. Another option is "[foo bar baz]", but that doesn't seem to be easily readable as "the slice of foo from bar to baz". Plus, "foo[bar, baz]" is valid Python syntax and should probably be supported, although I haven't seen it used for much.



Compile-time Namespaces?

Macros have posed interesting challenges in terms of namespaces and scope. I think the difficulties have mostly been met. More exploration needs to be done, though. For example, would it be a good idea to allow importing at compile-time- that is, into the compile-time namespace which includes macros? You don't automatically get macro access like "(mymodule.mymacro foo bar)", since the form's first element is an expression, and you'd need to teach the compiler how to recognize when dotted attributes should be retrieved from modules at compile time and when they shouldn't. It seems initially like it would be a mess.

There are also questions about how to reference a shadowed macro: if I define my own macro called case, but I want it to expand eventually to the standard case macro, how do I get that? Do I have to bind it to another name every time I want to do something like that? Maybe that's not too bad. There certainly are times when it would be nice to have symbols bound to packages, though.

Tuesday, October 18, 2005

Shame! And, a Noodle reader

Okay, Jonathan linked here so I feel obligated to actually say something more useful here. I have neglected my webloggage duties.

I've been rewriting the Noodle reader in Python instead of the lex/yacc combo, so that I can build macro characters into it. It's more straightforward to read normal Lisp with a recursive one-item-at-a-time read.

The syntactic sugar added for Pythonicness makes things just a bit more complicated, though. I went through a few different attempts at a solution before I decided on implementing trailers (. or [], as in foo.bar or foo[bar]) using a new kind of macro character called a trailer macro. When a character follows another item immediately, with no intervening space, and it's set as a trailer macro, the handling function is called with the preceding item as a parameter, and the return value is used instead of the preceding item.

In the foo[bar] case, the reader would first read "foo", and just before returning, would peek at the next character in the stream. '['- oops- it's a trailer macro! So the read_subscript function is called with 'foo' as a parameter. The subscript "bar" is read. It's not a slice, so read_subscript returns a tuple we might express in Python syntax as (Symbol(subscript), 'foo', Symbol(bar)).

One problem was that the order of operations I came up with previously does not fit nicely here. It was really pretty complicated to make things separated based on precedence in the reader. Turns out that was a rather bad idea in the first place. I've also noted when writing Noodle that it's hard to remember the order of operations, since it comes into play so rarely. My solution is to have trailers always come first, then leaders (like the quasiquote backtick (`) or the tuple/symbol-quoting backslash (\). That makes everything unambiguous, is easy to remember, and is implementable without much headache.

One more thing I came across with this is the problem with * and ** being both symbols and leaders. It just isn't clean, and in the presence of reader macros is nearly impossible to correctly implement. So the leaders for varargs and kwargs are now @ and @@, and * is just another normal symbol-name character.

The new reader is much more versatile, and once I make it play nicely with string constants (correctly handling the backslash escapes, raw strings, and Unicode escapes) it will be ready to be dropped in to Noodle proper.

Thursday, July 14, 2005

Not Dead

Still doing some work on Noodle, in between other stuff, and it is slowly getting closer to a release. I'm not going to make myself write all the possible unit tests beforehand, and the docs will be minimal at release, so I've made the goals a bit easier to reach. Possibly others will help with the tests and docs and so on once it's released. Not that I'm depending on that--I'll still do it myself if necessary, since this project is interesting enough to me.

I do want to finish an implementation of eval-when before the release, though. That will clear up some things about what gets run when and hopefully keep the macros' environment from being confusing.

Monday, June 06, 2005

In Other News

I've been spending a bit of time on another marriage of Python and Lisp- embedding Python in sbcl, along with convenience functions for creating Python objects, accessing them, and so on. My friend Travis Hartwell started this by embedding Python using UFFI- enough to allow execution of a string of Python code- in only a few lines of Lisp. I was impressed and jumped in to help add stuff.

It's too early to say whether he'll want to release this publicly, or just make it a learning experience. We've got the distinct impression it's not of much interest to the Lisp community. Apparently the argument is that everything is so easy to do in Common Lisp, there is no need for a good comprehensive set of libraries and utilities. I don't know about that myself. Still, maybe other Pythonistas could make some use of it.

Tuesday, May 31, 2005

Progress Update

For the few people who already know about Noodle and are waiting for the release: It doesn't look like I'll make the expected June 1st date. But over the holiday weekend (and in the course of a road trip to see in-laws in Sacramento) I was able to get a lot of good work done on Noodle.

There were some elusive problems caused by assumptions made by Python bytecodes. See, in Python, the stack is empty between each statement. But in Noodle, everything is an expression, so sometimes there will be things on the stack when they start. Most of the time, this doesn't cause a problem. But with things like Python's CONTINUE_LOOP and BREAK_LOOP, it makes big problems. They both simply wipe out the stack when called.

So, Noodle's continue and break special-operators [1] associated with loops now keep track of the stack state when they are created, and when called, POP_TOP items off the stack to get to that point. They also keep track of exception blocks and pop them off (POP_BLOCK) when appropriate. Finally, loops are implemented without the SETUP_LOOP construct, since it's only useful for making the CONTINUE_LOOP and BREAK_LOOP ops work. So continue and break have to manually jump to the correct place. For all this to work, Noodle needs to keep much stricter track of the exact predicted stacklevel for each individual opcode than it did before. But tightening that up has helped me to find one or two extra bugs that are now fixed. And these changes should make it simple to add support for things like (break 2) or (break-myloop) for breaking out of loops other than the innermost.

Distutils-ization is done, and there are more standard macros, and the Bison/Flex parser and scanner are working wonderfully. Along with that, compilation is faster, the error messages for failed parses and scans are more descriptive, and the lnotabs (line number tables) are much more accurate.

So I'm closer to having Noodle ready for a release (by which I mean, Noodle is almost to the point where I'm not as worried about having everyone mock me when they see it :).

[1] I'm not sure whether continue and break should be considered special operators, since they are defined and put into the macro dict when inside loops, by the special operators for loops. But they are defined in Python and are otherwise just like special operators.