What can I say? The Greatest Language Ever and the Greatest Language Possible are both doomed propositions. The former ignores the infinite variation of use cases, platforms, coding styles, and the sociology of individuals and groups. The latter does all of that and goes still further by arbitrarily choosing language features for which the "best" is likely to be a highly debatable matter. Like all good philosophical questions, these one's don't have concrete answers. We pose them not to arrive somewhere, but to stimulate thought and further innovation.

But Lisp is still the Greatest Language Ever.

I only learned Lisp a couple of years ago. I'm not sure how I got there, really. I think I might have stumbled on one of Paul Graham's essays (perhaps from a Slashdot interview Kent Pitman gave some time ago). I was thoroughly intrigued by what Graham had to say, and decided to give it a shot. That, of course, led me into the Scheme vs. Common Lisp debate, as well as to the question of which implementation to use.

I think it took me about an hour to decide against Scheme. Nearly everything I read about it suggested that it was a highly minimalistic version of Lisp intended for introductory computer science courses and ponderous experiments in the study of languages and algorithms. In other words, it wasn't meant for serious application development at all. In fact, the very first Q from the Scheme FAQ sealed it:

Scheme is often used in computer science curricula and programming language research, due to its ability to represent many programming abstractions with its simple primitives. Common Lisp is often used for real world programming because of its large library of utility functions, a standard object-oriented programming facility (CLOS), and a sophisticated condition handling system.

So (I concluded), if you want to build real apps you want Common Lisp. I ended up going with SBCL (which is a fantastic implementation). I also read Peter Seibel's book, which is really sensational.

I started this recent series of blog posts, because I wanted to talk about why I'm both dazzled and frustrated by certain aspects of Common Lisp. None of my objections involved the language itself (I'm not even sure I follow some of the critiques of CL that show up on the newsgroups). My problems were the garden variety ones. CL folks use Emacs/Slime. I use Vim (which has very weak support for IDE-style development with Lisp). And anyway, I hate IDEs. I want the compiler to compile (or the interpreter to interpret), the debugger to debug, the editor to edit, and I want to switch tools whenever I want. I also don't want anything abstracting me away from the details of the tools themselves. That's just me, but there it is.

I loved having a REPL, but I hated having just a REPL, and while it's possible to get SBCL to work with a shebang, it involves a lot of hacking of configuration files. Package installation was often complex and tedious, and the library I was looking for usually wasn't available (or if it was, it had absolutely no documentation whatsoever). I fully understood the arguments about why it's hard to create standalone "binaries" for CL, but I wondered why there wasn't something like a Java jar file — some easy way to launch an app with minimal fuss.

As I've said before, there is a way to do all of this if you're patient. But when you read the posts about these matters, you witness a very peculiar rhetorical trend. Someone posts asking about, say, how you go about creating an "executable" with Clisp or SBCL. Someone might respond with a howto, but the person asking is much more likely to get a sermon about how they need to free their mind. They're told about how the line between development and use is blurred in Lisp, about how "standalone programs" are actually a myth, since the operating system ships with the C runtime and is therefore providing you with mountains of scaffolding (no different from Lisp!). They're told that the REPL is a perfectly legitimate "shell" for ordinary users. And anyway, the UNIX shell is a horrific hack from which we should all attempt to extricate ourselves. In fact, it's really much better to have Lisp insulated from the fallen world of shells, C libraries, conventional interfaces, etc. You just need to embrace the Lambda Nature.

Yeah. Whatever.

I get it! I really do. I understand these arguments. In fact, some of my objections are completely unfair (No documentation? No libraries? Well surely you can do something about that!). Other points are absolutely dead on (like the one about the C runtime). It's just that I'm trying to get my work done and make my users happy. They will not be happy if they have to fiddle with the common-lisp-controller, set up aliases for asdf, recompile fasls, and then learn Lisp syntax so they can use my program. And I'm going to be unhappy if I have to wade through source code for lack of documentation, struggle with the FFI, and spend an hour setting up my system so I can use a particular package. It may be — in fact it is — the Best Language Ever. But it's not the Best Userspace Tool Ever. Not by a long mile.

So I started looking around for other Lisp alternatives, and this led me to a long, second look at Scheme.

I discovered that the opening salvo of the FAQ is absolutely true. It is a very small language, and is therefore ideal for teaching and language research. But what the FAQ was leaving out, I think, is the startling implications of a language that is trivially easy to implement (relatively speaking). If you're going to implement CL, you better lock and load. It's going to be a major job. Scheme? Well that's a different story.

The fact that Scheme is easy to implement has led to dozens and dozens of implementations. Some of them are "pure," in the sense that they provide the core language and don't concern themselves with OS integration. That's perfect if you're teaching Lisp. But what if you want to target the JVM ("for shame!" say the CL folks)? Or compile to C? Or run it on a Palm? Or use it as a systems programming language? Or embed it in another program? Or embed it in another programming language? Or put in on a cell phone? Or use it as the scripting language for your editor?

And we can go further. What about a Scheme that is designed to work seamlessly with UNIX? Or a Scheme that makes it trivially easy to call native libraries? Or a Scheme that can be used as a shell? Or a scheme that acts like an ordinary "scripting language?"

What I discovered, is that there are Scheme implementations for all of that. They're all Scheme (and therefore all Lisp), but none of them try to be all things to all users (or worse, impose philosophical restrictions on how the language is used).

Imagine my delight when I stumbled across Chicken Scheme (one of several that try to fill in a particular programming niche). This is a Scheme implementation that compiles to C (using a very clever method), which means that you can very easily create the elusive "executable" with a single command, and it will behave like any other native app. Its package system is brainlessly easy to use, and it has lots of libraries to do good-old-fashioned every day stuff (talk to databases, munge XML, export to CSV, etc.). In fact, it really feels like Ruby or Perl or Python. But it's fast! And it's Lisp!

That was enough to make me fall in love, but my real moment of epiphany came when I discovered Easyffi.

All of the CL implementations allow you call native functions through a foreign function interface. I read over some of the specifications for doing that (CFFI and UFFI). "Yeah, it's doable," I thought, "But it's going to require some serious effort."

Here's how you call native C libraries from Chicken using Easyffi:

1. You reference the C header in your source file.
2. You call the native function (in Scheme).

Sorry, did I miss something? I mean, it's a little more complicated than that, but not much. I sat down to write an interface to the native GraphViz library. I had it working in about a half an hour.

The mind reels. I'm speechless. So, I can just create interfaces to any C library I want (making them as Lispy as I care to with macros and whatnot)? Just by calling them? And then I can compile the programs that call those interfaces into fast C code that will behave like an ordinary POSIX program?

And I can do all this in the Greatest Language Ever!?

I'm a convert. I've known for awhile that Lisp is the red pill. Now it seems that, for me at least, Chicken Scheme (and Scheme in general) might be the little blue one.

I ended my last post, The Greatest Language Ever, with a personal conundrum. Why is it that, given a project for which the choice of language was not influenced by extraneous factors, did I choose Ruby rather than Lisp?

Anyone who has lingered around the Lisp community would probably be able to see the answer coming: lack of libraries, no easy way to get a standalone binary, not enough momentum behind the language, the loneliness of being the only hacker in sight using Lisp, lack of integration with POSIX, etc., etc. And the Lisp fans have various answers prepared for all of these. If you go over to comp.lang.lisp and search for the keywords in the last sentence, you can see the whole battle. In fact, you can peruse at least a decade's worth of the same questions and answers going back and forth.

The short answer is usually that Common Lisp does all of these things. In other words, there are many excellent libraries, you can create standalone programs, Lisp is going to undergo a great resurgence any day now (have you read Peter Seibel's excellent book? It won a Jolt Award!), and you can integrate with POSIX. But the situation — my situation and the situation of many others — is more complicated than that. In fact, I think both sides are right.

But rather than pick through the claims and counterclaims, let me step back for a moment and talk about The Greatest Language Possible.

If the Greatest Language Ever is a ridiculous notion (for the reasons outlined in my previous post), the Greatest Language Possible is even more so. Yet in a way, the idea is even more irresistible. Frederick Brooks was right to note, in The Mythical Man-Month, that programmers are optimists. We will vastly underestimate the time it takes to do something, naively declare things to be easy or even "trivial" to implement, and generally approach most problems, to our extreme peril, with a can-do attitude. When we talk about dream languages, we're really expressing a longing for the perfect tool — the one that really would make everything easy and trivial.

I mentioned that my favorite languages are C, Ruby, and Lisp, and I think my perfect language would blend all three.

A few years ago, I posted the following on Slashdot. It was essentially a response to a critique of C:

You know, I never hear anyone disagree with the type of thing you're saying here. It's unsafe, there's no garbage collection, the pointer abstraction is confusing, the macro system is terrible, it's just high level assembler, etc., etc.

But there are lots and lots and lots of people who code Linux programs in C — the vast majority of the programs I come across, at any rate. These people surely aren't being coerced into it by some manager. I assume that most of the people writing sophisticated software are multilingual (hard to get to get really good without learning at least half a dozen languages, in my opinion). They surely all know about OO, templates, assertions, abstract classes, and whatever other language features are out there.

It's like the biggest silent majority in computing. All these people merrily hacking away in C without complaint.

I would like to hear someone talk about why they like to code in C. I'm not asking for a language war. I would just like to hear from this silent majority.

The first response back said it all:

C is for people who understand how computers work, and how they work best, and who can't or won't accept "easier" or "faster" (to develop in) programming languages just because they won't always generate the best code.

Now, them are fightin' words, for sure, but I think I agree. I am one of those who has been merrily hacking away in C (nothing big, just stuff for my own system) for years without a complaint. And I think I love it because of its primitive nature. There's something about programming on the metal — thinking about memory usage, imagining the content of the registers, breaking open every black box — that is appealing in some basic way. But there's also the practical matter of being a UNIX programmer. C is the common tongue of UNIX. And it's fast. Really, really fast.

Ruby is a joy for exactly the opposite reason. It keeps you blissfully unaware of the details. There are even times when you feel like you're speaking English to the computer. It's elegant, "fast enough" in most cases, and there are lots and lots of libraries that make it easy to talk to native C interfaces. In a way, I always want the library to be in C, but at the same time, I always want to call the library in Ruby. It's even pretty Lispy, as non-Lisp languages go, and includes some direct borrowings (though, as an aside, I find it interesting that I rarely use lambda functions when writing Ruby, and I think it's because the Algol-like syntax doesn't lend itself to this kind of expression. The parentheses in Lisp aren't just there for decoration.)

And then there's Lisp. So elegant and expressive, so much fun to write. Half the time, I don't know whether to laugh or cry.

Somewhere in here is my ideal. Not my metaphysical pronouncement on languages, but the tool I wish I had at my disposal. It's as fast as C, yet as high as Ruby. It speaks UNIX effortlessly, and it's trivial (there's that word again!) to call native libraries from it. It has a vast library (perhaps as a function of it being so tightly integrated with C), and it's reasonably portable. Oh, and it's also a Lisp.

I chose Ruby over the Greatest Language Ever for the recent project not because of any show-stopping defect in the Lisp implementations or overwhelming virtue in Ruby, but because you have to weigh a complex set of factors for the particular problem and hope for the best. Lisp has the language, but Ruby has the libraries. It would be possible to hook Lisp into the libraries, but that's a complicated matter, and it would constitute a clear case of overengineering just to get some proof-of-concept code working. Writing those libraries would be the patriotic thing to do, but Lisp's FFIs are complicated and I don't have time to do it. None of this is the fault of Lisp, Lisp's implementors, Lisp's advocates, Lisp's community, or anything else. It's just a hacker on a deadline.

But did I mention that I think I've found an implementation of Lisp that does all the things I've been talking about? Could it be the dream language?

Be still my beating heart. Next post.

Over the years, I've managed to learn a number of different programming languages. I'm not an expert in all of the languages to which I've been exposed, but I've written at least one nontrivial program (and in most cases, several such programs) in C, C++, Perl, Python, Java, PHP, Common Lisp, Scheme, Prolog, and Ruby. Broaden the definition slightly, and I suppose you could also include Bash, Awk, and SQL.

There are a lot of factors involved when making a decision about which language to use for a particular project, and very few of them have to do with a language being the "best language" as such. C is an excellent language for writing mouse drivers; it's probably not the best choice for AI. Java is a miserable language for writing operating systems; it's quite a good choice for server-side web applications. But of course, even that is a largely subjective matter. Plenty of people do low-level systems programming (of sorts) in Java, write GUI applications in C, and do AI with PHP (okay, maybe not). And anyway, this isn't even the biggest factor. You pick a language because your "group" (your company, your project, your school) uses that language, because it's the one you know best (or the only one you know), because there's a fantastic library available for that language that fits perfectly with what you want to do, and so forth.

The question of which language is best is also full of minor religious questions. Static typing or dynamic typing? Interpreted or compiled? Object oriented? Functional? Procedural? Garbage collected? Since it is provably possible to write anything computationally tractable in any language, one finds oneself reduced to a balanced tri-fold truism about programming languages:

1. All languages (approaches to compilation, syntax, etc.) have advantages and disadvantages
2. The "best language" is likely to be the one that is most suited to the particular task at hand
3. Javascript is the worst language ever

But really, that's no fun at all. It may be true that questions about which language is best are mostly subjective, but if you write software for a living, your own likes and dislikes begin to seem like truisms themselves. Surely the cabinetmaker doesn't need to justify her love of a particular hand plane?

Of all the languages on my list, there are three that I genuinely love: C, Ruby, and Lisp. And the greatest of these is Lisp. In fact, I'll just say it: Lisp is the Greatest Language Ever. Period.

The reasons why Lisp is the Ultimate Language have been eloquently expressed elsewhere, so I won't rehearse the technical merits of the language (particularly since, as I just noted, such rehearsals are futile). I will say, though, that it radically changed the way I think about programming (in any language).

My experience with Lisp was something akin to my father's experience with Wagner several years ago. He had been listening to opera for many years, owned a vast collection, and had become quite an "expert listener," but hadn't purchased any Wagner. So he bought a copy of Das Rheingold. He tried to like it, but the first few listens left him cold. In fact, he really thought it was awful (despite the fact that many of his fellow opera afficianados consider it possibly The Greatest Opera Ever written). After awhile, though, he found that several passages were beginning to grow on him. In fact, he started to really like Das Rheingold. So, he bought the second one (Die Walküre). Same thing. Sounded like noise.

By the time he finished, he told me that he had to put away Der Ring des Nibelungen because, as he put it, "It was making the rest of my opera collection look amateurish. It's truly the greatest opera ever."

My experience learning Lisp and coding a few serious programs in it was deeply profound. Afterward, most of the other languages I had been using regularly seemed silly and inane. I found myself saying things like, "I've spent an hour on this method, and I know I could knock it off in three lines of perfectly elegant Lisp!" And what I was lacking in those languages was a whole host of programming constructs I had either never heard of or only dimly understood before learning Lisp. Passing a function into a function? Why would you ever want to do that! You wouldn't. Unless you got seriously into Lisp, in which case you find yourself (a) doing it all the time and (b) writing much more elegant, expressive code as a result. Same goes for closures, continuations, anonymous functions, and macros.

Macros, in particular, might be the most dazzling thing I've ever seen in programming.

Programming is all about "extending the language." Any language will have hundreds of built-in functions, and when you write an application, you are essentially using those functions to write new ones for your own application. But there are major limits to this. You might be able to write a function called "read_data_stream()" or "sort_pyramid()," but you can't extend the language at more fundamental levels. You can't, for example, write your own if-then-else block or for-loop (and have it work like a part of the language itself). In Lisp, you can — and you find yourself doing it effortlessly after awhile. In fact, you find yourself using Lisp to write the programming language you wish existed for your particular problem. And with Lisp, that is a thousand times easier than it sounds. Do that for a few weeks, and all the other languages will seem like crude stone tools.

I've spent the last week or so writing in Ruby. It's a wonderful language. It's a joy to program in it, and I'm almost always trying to find ways to overcome the various reasons not to ("that's not the language of the project," "there aren't enough libraries," etc.). I know it like the back of my hand, and I teach the language every year. It even has a number of Lispy constructs (like closures, continuations, and anonymous functions). But it ain't no Lisp. In fact, the only way it could be as good as Lisp would be for it to be a Lisp dialect (which it is not).

But last week, I was writing proof-of-concept code. There was no one telling me what language to use, no "audience" for the code, and there were good, serviceable libraries available for Common Lisp, Ruby, Java, Python and Perl. I went with Ruby.

The question, of course, is why not write it in The Greatest Language Ever?

Well, that's a separate rant, which I'll save for another post.

I have for many years searched high and low for an electronic whiteboard app. It's one of those Great Ideas that many have had, but every implementation I've ever tried has been terrible (slow, clumsy, feature poor — usually all three). It never seemed like that difficult a thing to create, but I've never used one more than once.

Until now.

Patrick K. over at the Z-Axis tipped me off to skrbl — the first fast, usable, feature-rich whiteboard app I've ever seen. Patrick has a gushing review that summarizes both my feelings about past attempts at whiteboard nirvana and my elation at the joys of using skrbl.

Of course, I still can't draw . . .