Back in 2003, at the Digital Humanities conference in Athens, Georgia, Geoffrey Rockwell and I threw all caution to the wind and performed a live Brechto-Socratic dialogue on the relationship between programming and writing. People still ask us about it, but we've never been sure exactly what to do with it.

The intellectual glow of THATCamp must have put me in the mood, because this afternoon it came to me — the ideal forum for what Geoffrey and I privately refer to as "Untitled #4."

It was impossible to give Geoffrey's character his signature beard, but other than that, I would say it perfectly recreates the original performance (including our extraordinary range as "voice talents"). Enjoy!

You can also see it here and here. Or, um, here.

I will admit to being a huge (read, fawning) fan of Paul Graham. I've never met him, but I think his essays are terrific (especially the ones on Lisp). He clearly has a ninth-degree black belt in programming, and yet he's one of the most clear-headed prose writers I've read when it comes to dilating complicated technical subjects. I, personally, would have no idea how Lisp macros — the single most powerful and mind-bending concept I've encountered in programming — work without the benefit of his book On Lisp, which has been circulating freely as digital samizdat on the Web for many years. Since I aspire to be both a black belt programmer and a clear-headed writer, I tend to take what he says and does very seriously indeed.

I'm not alone, of course, and so when Graham announced that he was creating a new dialect of Lisp (called "Arc"), I and many others got very excited. We were all particularly intrigued by the statement of design philosophy that appeared on his home page a couple of years ago, and have been waiting with bated breath ever since.

Well, Arc is out. And with its release (more of a stable pre-release) came a torrent of criticism. In fact, the criticism began years ago (Arc is vaporware, etc.). I won't rehearse the criticism here; I'll just make the general point that people have a lot of damn gall.

As far as I can tell, Paul Graham's reasons for writing Arc are the same as for any voluntary development project. He'd like to give himself the tool he wishes he had, he'd like to amuse himself intellectually, he'd like to explore various aspects of design, he'd like to play around with some ideas he's had for years, and he'd also like to contribute something useful to the world. These seem to me excellent reasons for doing just about anything, and it's nice that he's chosen to throw in the last one.

There are several legitimate reasons to feel miffed about software. You might be frustrated with the release schedule or the number of bugs or the fact that such-and-such a feature isn't implemented yet. You might find the community that supports it unhelpful or the authors arrogant and imperious toward their users. You might think the whole thing is wrong from the start.

If it's a commercial piece of software for which you've paid money, I can understand angry charges and criticisms. But FREE (as in beer, as in freedom, as in range, whatever) software? What gives anyone the right to shoot their mouth of like this?

It's not that people can't have these opinions. It's that they air them so freely and with so little charity. Graham would be well within his rights to take his toys and go home, after all. He wouldn't be the first FOSS developer to do so. It seems like once a month some talented hacker writes a farewell letter in which he or she admits that the constant battering is starting to wear them down.

I'm not suggesting that people stop critiquing software and design philosophies. Nor am I suggesting that everyone suppress their natural frustrations. I'm merely suggesting that all such critiques and frustrations be firmly wound up in a gentle cloak of charity, good will, and constructiveness. It might be true that the developer can tell a jackass from a contributor, but he or she might not. And the consequences of beating up on people in this way is the suppression of the motives I outlined above. Do we really want a world in which people don't play for love of the game, but only because they're compelled to by some other force?

In that piece on design philosophy, Graham wrote, "The great languages have been the ones that good programmers designed for their own use– C, Smalltalk, Lisp." I think he's right about that, and I think we could probably generalize that sentiment to lots of areas of human endeavor. Fortunately for us, lots of people with that idea have drawn the perfectly humane conclusion that what's useful to them might be useful to others. If drawing that conclusion brings you nothing but grief, there's no loss in really making it something for your own use. After all, couldn't you just work on the project for love of the game and keep it to yourself?

If I were Graham, I might be asking myself why I'm even bothering. And that bothers me.

Stéfan Sinclair, one of DH's most talented hackers, responded to the last post with a question about the "dominant language" in DH. I suggested it was Java, but (as I noted in the comment thread), I really have no basis for saying that. Here's Stéfan:

I’d say (as unempirically) that there are more DH projects developing (not just using) code with scripting languages like PHP, than there are with Java, especially outside of the larger centres. But maybe this is just speculation based on what I see as the norm for project cycles: a researcher (or small team) gets some funding which includes money for hiring some research assistants – often graduate students – who are more likely competent in PHP (or Ruby or Perl) than in Java. Moreover, I think DH projects tend to favour getting something up fairly quickly over design and robustness; another reason why scripting languages would be more prominent. There are centres with dedicated staff willing and able to work in Java, but that seems to me a relatively rare luxury.

I didn’t mean to nitpick regarding a minor part of the post, I was just curious. I don’t think it’s crowning a champion programming language that matters, it’s more about how the reality of the DH research environment should be influencing the curriculum (or perhaps there are too few of us teaching programming for it to matter that much).

I think Stefan is probably right, but of course, both of us are relying on general impressions.

When you look at the development of computer science curricula over the last few decades, you can clearly see that discipline responding to industry pressures. Scheme might remain the language of Hal Abelson's venerable 6.001 course at MIT, but in general, we've watched CS go from Pascal, to C, to C++, to Java as the demand for engineers with particular "skills" has changed out in the real world. I don't know many CS professors who think C++ and Java are good teaching languages, though. Many of the professors I've talked to actually long for the days of Pascal, but they also know that students will object to learning a language that isn't in common use.

When I started out in programming, I asked a friend (a very skilled hacker who was pursuing a Ph.D in CS at the time) what language I should learn first. I remember very clearly his answer, which was something like, "Oh, I don't know. Just pick one. It doesn't matter. How about a fake one? There are lots of cool pseudo-languages out there that will teach you what you need to know . . ." I thought he was 100% insane, and I was astonished that this guy — who was and is a master programmer, in addition to being a highly skilled theorist — would suggest something so obviously out of touch with reality. A few years later, I realized that he was 100% right. What's important in programming is the concepts. If the goal is to learn programming and software engineering, the language literally doesn't matter. Abelson's Structure and Interpretation of Computer Programs (the textbook for 6.001), not only uses Scheme, but avoids explicitly teaching the language, per se. It's perhaps the best book I've ever read on programming.

DH, thankfully, is not burdened with "industry pressures" in the way CS is. When I tell my students that it's the concepts that are important, they believe me. They believed me even when none of their programming friends knew what Ruby was. I think they'd still believe me if I suggested that we all learn Haskell or Miranda. And that's a good thing.

Still, it would be very useful to know what languages people regularly work with in DH. If we knew the answer to that question, those of us who teach programming for the Humanities (and I know there are only a few of us out there) could perhaps structure the teaching of the concepts in such a way as to make it easy to transfer that knowledge into other kinds of languages. I do this a little bit already, by occasionally pointing out the ways in which different languages — like C or Javascript — approach some concept that we're studying in Ruby. I do that in part to emphasize that the fundamental concepts of programming don't change drastically from language to language.

It would also be good to know what languages people use in DH, because it might help us to focus development where it's most needed. We've been talking quite intensively about "tools" over the last few years in the DH community, but I've always felt that "tools" might best be thought of as an alias for libraries and APIs.

It's not possible to do a scientific poll on a blog like this, but I suspect my readership has enough hackers in the ranks to get some good anecdotal information.

So how about it? What languages do you use? What languages do the people around you use? What languages are people telling you you should know? I'd love to hear about it!

Well, it's time to pick up another programming language.

I've been doing this about once a year since the late nineties, and in that time, my motivations for programming glossolalia have changed a few times. At first, I think I mostly wanted to jump on the latest bandwagon. I was also highly susceptible to arguments from more experienced hackers about how such-and-such a language was the Greatest Thing Ever. But after awhile, I found myself wanting to study languages because I find them completely fascinating. Nowadays, I think it's just a way to expand my thinking about what languages are and what they can do.

But honestly, after doing this ten or twelve times, I find that I'm beginning to stray into the outer rim. What's next? Haskell? Erlang? I had more or less decided that I'd do one of those over the summer, but a couple of days ago I stumbled on Groovy.

What's Groovy? A language with an absurd name, for one. But beyond that, Groovy (according to the home page):

• is an agile and dynamic language for the Java Virtual Machine
• builds upon the strengths of Java but has additional power features inspired by languages like Python, Ruby and Smalltalk
• makes modern programming features available to Java developers with almost-zero learning curve
• supports Domain-Specific Languages and other compact syntax so your code becomes easy to read and maintain
• makes writing shell and build scripts easy with its powerful processing primitives, OO abilities and an Ant DSL
• increases developer productivity by reducing scaffolding code when developing web, GUI, database or console applications
• simplifies testing by supporting unit testing and mocking out-of-the-box
• seamlessly integrates with all existing Java objects and libraries
• compiles straight to Java bytecode so you can use it anywhere you can use Java

I've used a few alternative languages for the JVM, including JRuby, Jython, and Kawa. I mean, what's not to like? The syntax of Ruby, Python, or (be still my beating heart) Scheme with all the might and magic of the Java class library? What could be better?

In my experience, the original languages and runtime environments are better. I don't know what it is. I always feel like the Javish stuff just don't belong there somehow — like it's some kind of crude hack. And before I get hate mail, let me say that these implementations are not crude hacks. They're very skillfully done. There's just something about the mixture that doesn't sit well with me.

I haven't started looking closely at Groovy yet, but I can see the general plan. They've taken some of the best ideas from languages like Ruby, Perl, Python, and Smalltalk, kept the general contours of Java's C-ish syntax, and built a scripting language that has all the soul-stirring power and ubiquity of the Java class library in a nice little package. And, of course, it's trivially easy to embed Groovy in Java or to integrate them in some other way.

Now, I might find it annoying (that is: slow, or poorly documented, or whatever). But if it really does have the ease of use and expressive power of a good scripting language, I may start teaching it in my Digital Humanities courses. I've been teaching Ruby to English majors for a number of years now, and I still think it's a magnificent teaching language. But if Groovy was similarly good for teaching and allowed an easy migration path for students interested in Java, I just might be sold. Java is the dominant language in DH, and while a few of my students have been able to pick up Java after learning Ruby, it would be nice to have a language that was semantically closer to Java. I could imagine a class that goes through Groovy, and then ends by starting off with Java. But then, Groovy really would have to be groovy.

They will have to change the name, though. Can I really have a course description that says we'll be learning Groovy?

I'll report back when I've played with it some more.

Non-programmers are often bemused to hear programmers speak of beauty and elegance in code. It's a strangely ineffable assignation, yet one that every serious programmer can make. Some pieces of code are not just utile, but dulci. And nothing thrills the heart of an engineer like the moment of beauty.

Paul Graham offers a code example in his book On Lisp (out of print but available on line) that strikes me this way. It's just a tiny little "counter" procedure:

(defun make-adder (n)
   #’(lambda (x) (+ x n)))

This function takes a number, but instead of returning an incremented value, it returns a closure that will add that number to any subsequent function call. Hence:

> (setq add2 (make-adder 2)
>         add10 (make-adder 10))
> (funcall add2 5)
> 7
> (funcall add10 3)
> 13

Graham calls this an example of a "utility function," but I must admit that when I first saw it, my reaction wasn't "Wow, what a useful thing!" or "Ah, now I understand how lexical closures work!" My reaction was simply, "Good heavens, that's gorgeous."

But why? Perhaps it's some convergence between beauty and cleverness. Cleverness isn't always a virtue in programming (particularly since such moments often exchange clarity for concision), but sometimes things manage to be both clever and beautiful — clever because they're beautiful, and beautiful because they're clever. And "cleverness" is often (though not always) a component of art. The Mona Lisa exudes a kind ineffable cleverness, as does the astonishing photorealism of Old Masters paintings, and in both cases, it's not merely craft that produces this sensation.

Beauty in code is not a function of utility at all (though the two often accompany one another). This code sample (found both in Sitaram's Teach Yourself Scheme in Fixnum Days and Dybvig's The Scheme Programming Language) also strikes me as beautiful. It's a set of recursive co-routines for deciding whether a number is even or odd:

(define is-even?
   (lambda (n)
      (if (= n 0) #t
         (is-odd? (- n 1)))))

(define is-odd?
   (lambda (n)
      (if (= n 0) #f
         (is-even? (- n 1)))))

These co-routines pass the number back and forth; you get your answer, because the function that returns the answer will be the last procedure called — the last player with the football, as it were.

In one sense, the code is perfectly practical, since it's prefatory to a discussion of letrec — a variety of conditional that allows you to assign local variables recursively. But in another sense, it's entirely silly. Scheme, after all, provides both odd? and even? procedures. But ah, what a beautiful piece of code. It's not, "Wow, isn't Lisp great!" or "Ah, now I understand letrec!" It's just "What a beautiful piece of code." For me, at least.

Hand a geek a pocket watch, and he or she will immediately open up the back and stare at the mechanism for an hour. And if you could persuade that person to speak out loud, they'd be saying things like, "Look at that! And that! Wow, that's just . . . beautiful!" And that really is something like an aesthetic experience that is in a way indistinguishable from the experience of art.

I don't mean to argue that Lisp is inherently beautiful; the examples are in Lisp because that's my current obsession, and I'm reading a lot of Lisp code these days. In fact, I think what I'm talking about here transcends language. It's certainly not a matter of syntax. One could easily imagine beautiful C, Java, or Fortran. I had certainly seen lots of examples of beautiful code before I knew anything at all about Lisp.

Got some beautiful code (in any language)? I'd love to hear about it.

What can I say? The Greatest Language Ever and the Greatest Language Possible are both doomed propositions. The former ignores the infinite variation of use cases, platforms, coding styles, and the sociology of individuals and groups. The latter does all of that and goes still further by arbitrarily choosing language features for which the "best" is likely to be a highly debatable matter. Like all good philosophical questions, these one's don't have concrete answers. We pose them not to arrive somewhere, but to stimulate thought and further innovation.

But Lisp is still the Greatest Language Ever.

I only learned Lisp a couple of years ago. I'm not sure how I got there, really. I think I might have stumbled on one of Paul Graham's essays (perhaps from a Slashdot interview Kent Pitman gave some time ago). I was thoroughly intrigued by what Graham had to say, and decided to give it a shot. That, of course, led me into the Scheme vs. Common Lisp debate, as well as to the question of which implementation to use.

I think it took me about an hour to decide against Scheme. Nearly everything I read about it suggested that it was a highly minimalistic version of Lisp intended for introductory computer science courses and ponderous experiments in the study of languages and algorithms. In other words, it wasn't meant for serious application development at all. In fact, the very first Q from the Scheme FAQ sealed it:

Scheme is often used in computer science curricula and programming language research, due to its ability to represent many programming abstractions with its simple primitives. Common Lisp is often used for real world programming because of its large library of utility functions, a standard object-oriented programming facility (CLOS), and a sophisticated condition handling system.

So (I concluded), if you want to build real apps you want Common Lisp. I ended up going with SBCL (which is a fantastic implementation). I also read Peter Seibel's book, which is really sensational.

I started this recent series of blog posts, because I wanted to talk about why I'm both dazzled and frustrated by certain aspects of Common Lisp. None of my objections involved the language itself (I'm not even sure I follow some of the critiques of CL that show up on the newsgroups). My problems were the garden variety ones. CL folks use Emacs/Slime. I use Vim (which has very weak support for IDE-style development with Lisp). And anyway, I hate IDEs. I want the compiler to compile (or the interpreter to interpret), the debugger to debug, the editor to edit, and I want to switch tools whenever I want. I also don't want anything abstracting me away from the details of the tools themselves. That's just me, but there it is.

I loved having a REPL, but I hated having just a REPL, and while it's possible to get SBCL to work with a shebang, it involves a lot of hacking of configuration files. Package installation was often complex and tedious, and the library I was looking for usually wasn't available (or if it was, it had absolutely no documentation whatsoever). I fully understood the arguments about why it's hard to create standalone "binaries" for CL, but I wondered why there wasn't something like a Java jar file — some easy way to launch an app with minimal fuss.

As I've said before, there is a way to do all of this if you're patient. But when you read the posts about these matters, you witness a very peculiar rhetorical trend. Someone posts asking about, say, how you go about creating an "executable" with Clisp or SBCL. Someone might respond with a howto, but the person asking is much more likely to get a sermon about how they need to free their mind. They're told about how the line between development and use is blurred in Lisp, about how "standalone programs" are actually a myth, since the operating system ships with the C runtime and is therefore providing you with mountains of scaffolding (no different from Lisp!). They're told that the REPL is a perfectly legitimate "shell" for ordinary users. And anyway, the UNIX shell is a horrific hack from which we should all attempt to extricate ourselves. In fact, it's really much better to have Lisp insulated from the fallen world of shells, C libraries, conventional interfaces, etc. You just need to embrace the Lambda Nature.

Yeah. Whatever.

I get it! I really do. I understand these arguments. In fact, some of my objections are completely unfair (No documentation? No libraries? Well surely you can do something about that!). Other points are absolutely dead on (like the one about the C runtime). It's just that I'm trying to get my work done and make my users happy. They will not be happy if they have to fiddle with the common-lisp-controller, set up aliases for asdf, recompile fasls, and then learn Lisp syntax so they can use my program. And I'm going to be unhappy if I have to wade through source code for lack of documentation, struggle with the FFI, and spend an hour setting up my system so I can use a particular package. It may be — in fact it is — the Best Language Ever. But it's not the Best Userspace Tool Ever. Not by a long mile.

So I started looking around for other Lisp alternatives, and this led me to a long, second look at Scheme.

I discovered that the opening salvo of the FAQ is absolutely true. It is a very small language, and is therefore ideal for teaching and language research. But what the FAQ was leaving out, I think, is the startling implications of a language that is trivially easy to implement (relatively speaking). If you're going to implement CL, you better lock and load. It's going to be a major job. Scheme? Well that's a different story.

The fact that Scheme is easy to implement has led to dozens and dozens of implementations. Some of them are "pure," in the sense that they provide the core language and don't concern themselves with OS integration. That's perfect if you're teaching Lisp. But what if you want to target the JVM ("for shame!" say the CL folks)? Or compile to C? Or run it on a Palm? Or use it as a systems programming language? Or embed it in another program? Or embed it in another programming language? Or put in on a cell phone? Or use it as the scripting language for your editor?

And we can go further. What about a Scheme that is designed to work seamlessly with UNIX? Or a Scheme that makes it trivially easy to call native libraries? Or a Scheme that can be used as a shell? Or a scheme that acts like an ordinary "scripting language?"

What I discovered, is that there are Scheme implementations for all of that. They're all Scheme (and therefore all Lisp), but none of them try to be all things to all users (or worse, impose philosophical restrictions on how the language is used).

Imagine my delight when I stumbled across Chicken Scheme (one of several that try to fill in a particular programming niche). This is a Scheme implementation that compiles to C (using a very clever method), which means that you can very easily create the elusive "executable" with a single command, and it will behave like any other native app. Its package system is brainlessly easy to use, and it has lots of libraries to do good-old-fashioned every day stuff (talk to databases, munge XML, export to CSV, etc.). In fact, it really feels like Ruby or Perl or Python. But it's fast! And it's Lisp!

That was enough to make me fall in love, but my real moment of epiphany came when I discovered Easyffi.

All of the CL implementations allow you call native functions through a foreign function interface. I read over some of the specifications for doing that (CFFI and UFFI). "Yeah, it's doable," I thought, "But it's going to require some serious effort."

Here's how you call native C libraries from Chicken using Easyffi:

1. You reference the C header in your source file.
2. You call the native function (in Scheme).

Sorry, did I miss something? I mean, it's a little more complicated than that, but not much. I sat down to write an interface to the native GraphViz library. I had it working in about a half an hour.

The mind reels. I'm speechless. So, I can just create interfaces to any C library I want (making them as Lispy as I care to with macros and whatnot)? Just by calling them? And then I can compile the programs that call those interfaces into fast C code that will behave like an ordinary POSIX program?

And I can do all this in the Greatest Language Ever!?

I'm a convert. I've known for awhile that Lisp is the red pill. Now it seems that, for me at least, Chicken Scheme (and Scheme in general) might be the little blue one.

I ended my last post, The Greatest Language Ever, with a personal conundrum. Why is it that, given a project for which the choice of language was not influenced by extraneous factors, did I choose Ruby rather than Lisp?

Anyone who has lingered around the Lisp community would probably be able to see the answer coming: lack of libraries, no easy way to get a standalone binary, not enough momentum behind the language, the loneliness of being the only hacker in sight using Lisp, lack of integration with POSIX, etc., etc. And the Lisp fans have various answers prepared for all of these. If you go over to comp.lang.lisp and search for the keywords in the last sentence, you can see the whole battle. In fact, you can peruse at least a decade's worth of the same questions and answers going back and forth.

The short answer is usually that Common Lisp does all of these things. In other words, there are many excellent libraries, you can create standalone programs, Lisp is going to undergo a great resurgence any day now (have you read Peter Seibel's excellent book? It won a Jolt Award!), and you can integrate with POSIX. But the situation — my situation and the situation of many others — is more complicated than that. In fact, I think both sides are right.

But rather than pick through the claims and counterclaims, let me step back for a moment and talk about The Greatest Language Possible.

If the Greatest Language Ever is a ridiculous notion (for the reasons outlined in my previous post), the Greatest Language Possible is even more so. Yet in a way, the idea is even more irresistible. Frederick Brooks was right to note, in The Mythical Man-Month, that programmers are optimists. We will vastly underestimate the time it takes to do something, naively declare things to be easy or even "trivial" to implement, and generally approach most problems, to our extreme peril, with a can-do attitude. When we talk about dream languages, we're really expressing a longing for the perfect tool — the one that really would make everything easy and trivial.

I mentioned that my favorite languages are C, Ruby, and Lisp, and I think my perfect language would blend all three.

A few years ago, I posted the following on Slashdot. It was essentially a response to a critique of C:

You know, I never hear anyone disagree with the type of thing you're saying here. It's unsafe, there's no garbage collection, the pointer abstraction is confusing, the macro system is terrible, it's just high level assembler, etc., etc.

But there are lots and lots and lots of people who code Linux programs in C — the vast majority of the programs I come across, at any rate. These people surely aren't being coerced into it by some manager. I assume that most of the people writing sophisticated software are multilingual (hard to get to get really good without learning at least half a dozen languages, in my opinion). They surely all know about OO, templates, assertions, abstract classes, and whatever other language features are out there.

It's like the biggest silent majority in computing. All these people merrily hacking away in C without complaint.

I would like to hear someone talk about why they like to code in C. I'm not asking for a language war. I would just like to hear from this silent majority.

The first response back said it all:

C is for people who understand how computers work, and how they work best, and who can't or won't accept "easier" or "faster" (to develop in) programming languages just because they won't always generate the best code.

Now, them are fightin' words, for sure, but I think I agree. I am one of those who has been merrily hacking away in C (nothing big, just stuff for my own system) for years without a complaint. And I think I love it because of its primitive nature. There's something about programming on the metal — thinking about memory usage, imagining the content of the registers, breaking open every black box — that is appealing in some basic way. But there's also the practical matter of being a UNIX programmer. C is the common tongue of UNIX. And it's fast. Really, really fast.

Ruby is a joy for exactly the opposite reason. It keeps you blissfully unaware of the details. There are even times when you feel like you're speaking English to the computer. It's elegant, "fast enough" in most cases, and there are lots and lots of libraries that make it easy to talk to native C interfaces. In a way, I always want the library to be in C, but at the same time, I always want to call the library in Ruby. It's even pretty Lispy, as non-Lisp languages go, and includes some direct borrowings (though, as an aside, I find it interesting that I rarely use lambda functions when writing Ruby, and I think it's because the Algol-like syntax doesn't lend itself to this kind of expression. The parentheses in Lisp aren't just there for decoration.)

And then there's Lisp. So elegant and expressive, so much fun to write. Half the time, I don't know whether to laugh or cry.

Somewhere in here is my ideal. Not my metaphysical pronouncement on languages, but the tool I wish I had at my disposal. It's as fast as C, yet as high as Ruby. It speaks UNIX effortlessly, and it's trivial (there's that word again!) to call native libraries from it. It has a vast library (perhaps as a function of it being so tightly integrated with C), and it's reasonably portable. Oh, and it's also a Lisp.

I chose Ruby over the Greatest Language Ever for the recent project not because of any show-stopping defect in the Lisp implementations or overwhelming virtue in Ruby, but because you have to weigh a complex set of factors for the particular problem and hope for the best. Lisp has the language, but Ruby has the libraries. It would be possible to hook Lisp into the libraries, but that's a complicated matter, and it would constitute a clear case of overengineering just to get some proof-of-concept code working. Writing those libraries would be the patriotic thing to do, but Lisp's FFIs are complicated and I don't have time to do it. None of this is the fault of Lisp, Lisp's implementors, Lisp's advocates, Lisp's community, or anything else. It's just a hacker on a deadline.

But did I mention that I think I've found an implementation of Lisp that does all the things I've been talking about? Could it be the dream language?

Be still my beating heart. Next post.

Over the years, I've managed to learn a number of different programming languages. I'm not an expert in all of the languages to which I've been exposed, but I've written at least one nontrivial program (and in most cases, several such programs) in C, C++, Perl, Python, Java, PHP, Common Lisp, Scheme, Prolog, and Ruby. Broaden the definition slightly, and I suppose you could also include Bash, Awk, and SQL.

There are a lot of factors involved when making a decision about which language to use for a particular project, and very few of them have to do with a language being the "best language" as such. C is an excellent language for writing mouse drivers; it's probably not the best choice for AI. Java is a miserable language for writing operating systems; it's quite a good choice for server-side web applications. But of course, even that is a largely subjective matter. Plenty of people do low-level systems programming (of sorts) in Java, write GUI applications in C, and do AI with PHP (okay, maybe not). And anyway, this isn't even the biggest factor. You pick a language because your "group" (your company, your project, your school) uses that language, because it's the one you know best (or the only one you know), because there's a fantastic library available for that language that fits perfectly with what you want to do, and so forth.

The question of which language is best is also full of minor religious questions. Static typing or dynamic typing? Interpreted or compiled? Object oriented? Functional? Procedural? Garbage collected? Since it is provably possible to write anything computationally tractable in any language, one finds oneself reduced to a balanced tri-fold truism about programming languages:

1. All languages (approaches to compilation, syntax, etc.) have advantages and disadvantages
2. The "best language" is likely to be the one that is most suited to the particular task at hand
3. Javascript is the worst language ever

But really, that's no fun at all. It may be true that questions about which language is best are mostly subjective, but if you write software for a living, your own likes and dislikes begin to seem like truisms themselves. Surely the cabinetmaker doesn't need to justify her love of a particular hand plane?

Of all the languages on my list, there are three that I genuinely love: C, Ruby, and Lisp. And the greatest of these is Lisp. In fact, I'll just say it: Lisp is the Greatest Language Ever. Period.

The reasons why Lisp is the Ultimate Language have been eloquently expressed elsewhere, so I won't rehearse the technical merits of the language (particularly since, as I just noted, such rehearsals are futile). I will say, though, that it radically changed the way I think about programming (in any language).

My experience with Lisp was something akin to my father's experience with Wagner several years ago. He had been listening to opera for many years, owned a vast collection, and had become quite an "expert listener," but hadn't purchased any Wagner. So he bought a copy of Das Rheingold. He tried to like it, but the first few listens left him cold. In fact, he really thought it was awful (despite the fact that many of his fellow opera afficianados consider it possibly The Greatest Opera Ever written). After awhile, though, he found that several passages were beginning to grow on him. In fact, he started to really like Das Rheingold. So, he bought the second one (Die Walküre). Same thing. Sounded like noise.

By the time he finished, he told me that he had to put away Der Ring des Nibelungen because, as he put it, "It was making the rest of my opera collection look amateurish. It's truly the greatest opera ever."

My experience learning Lisp and coding a few serious programs in it was deeply profound. Afterward, most of the other languages I had been using regularly seemed silly and inane. I found myself saying things like, "I've spent an hour on this method, and I know I could knock it off in three lines of perfectly elegant Lisp!" And what I was lacking in those languages was a whole host of programming constructs I had either never heard of or only dimly understood before learning Lisp. Passing a function into a function? Why would you ever want to do that! You wouldn't. Unless you got seriously into Lisp, in which case you find yourself (a) doing it all the time and (b) writing much more elegant, expressive code as a result. Same goes for closures, continuations, anonymous functions, and macros.

Macros, in particular, might be the most dazzling thing I've ever seen in programming.

Programming is all about "extending the language." Any language will have hundreds of built-in functions, and when you write an application, you are essentially using those functions to write new ones for your own application. But there are major limits to this. You might be able to write a function called "read_data_stream()" or "sort_pyramid()," but you can't extend the language at more fundamental levels. You can't, for example, write your own if-then-else block or for-loop (and have it work like a part of the language itself). In Lisp, you can — and you find yourself doing it effortlessly after awhile. In fact, you find yourself using Lisp to write the programming language you wish existed for your particular problem. And with Lisp, that is a thousand times easier than it sounds. Do that for a few weeks, and all the other languages will seem like crude stone tools.

I've spent the last week or so writing in Ruby. It's a wonderful language. It's a joy to program in it, and I'm almost always trying to find ways to overcome the various reasons not to ("that's not the language of the project," "there aren't enough libraries," etc.). I know it like the back of my hand, and I teach the language every year. It even has a number of Lispy constructs (like closures, continuations, and anonymous functions). But it ain't no Lisp. In fact, the only way it could be as good as Lisp would be for it to be a Lisp dialect (which it is not).

But last week, I was writing proof-of-concept code. There was no one telling me what language to use, no "audience" for the code, and there were good, serviceable libraries available for Common Lisp, Ruby, Java, Python and Perl. I went with Ruby.

The question, of course, is why not write it in The Greatest Language Ever?

Well, that's a separate rant, which I'll save for another post.

I've created a 2-up version of why's poignant guide, which brings it down to about 63 pages.

I was an extremely early adopter of the Ruby programming language. In fact, I'm pretty sure I own one of the first printings of the first pickaxe book. There's even evidence to suggest that I was one of the first people to teach a course in Ruby at the college level. I'm not often so ahead of the curve, but there it is. I was way into Ruby way before it became fashionable.

And yet, somehow I managed to miss why's (poignant) guide to Ruby — an online tutorial on Ruby that comes close to being a graphic novel.

Thanks to Brett Barney of CDRH for pointing this out. He is obviously way, way, way ahead of the curve.