Terra Nostra

An Impression of Common Lisp

Written on 2018-12-04

Common Lisp is a lovely language to work with. Although it has faded into an unfortunate semi-obscurity on the modern computing landscape, it is still a powerful and elegant language that is a joy to use. I've had a lot of fun playing around with it and discovering more about it in the past two months, and wanted to record and share some of that in this post.

xkcd: lisp cycles

© Randall Munroe, xkcd.com

A very brief history of Lisp

Lisp is the second-oldest programming language still in use (developed in 1958 – only Fortran is older, by a few years). It had its heyday as the language of Artificial Intelligence research in the 60s to 80s.

Over the years, a large spectrum of dialects developed. Today, the three most prominent are Common Lisp, Scheme, and Clojure. The latter is quite new, built on top of the JVM, and has seen some recent popularity. Scheme is notable for being small and elegant, and is predominantly used as a teaching language (most famously at MIT, where it was used in conjunction with SICP).

Common Lisp itself was created in the 80s and 90s, with an ANSI standard published in 1994. Its purpose was to combine the features of various dialects then in existence, to bring a modicum of order into what had become a very fragmented landscape. Several implementations of the standard (both propietary and open-source) are available and still actively maintained. Though not particularly wide-spread in the world of software engineering, Common Lisp has carved out a niche for itself and retains a loyal following.

My history with Lisp

I first learnt Lisp in 2014. At the time, I had only been programming for two or three years (in Java and Python), and the very different approach used by Lisp really threw me. But after a couple of weeks of trawling through a good introductory book, I slowly got the hang of things.

My first bigger project was a framework for text-based adventure games (Project Atlantis), including an interpreter for a simple Domain Specific Language to describe game worlds. This was pretty much written in pure Common Lisp, with no additional libraries used.

The next few years I learnt regrettably little of the language, being occupied with other languages that I needed for university. Just recently, however, I had the chance to jump back into things – leading to the discovery that I had only ever scratched the surface of what Common Lisp can do. I learnt a lot more about the intricacies of the language itself, as well as how to work with the third-party package ecosystem. And that gave rise to this post…

Writing Lisp is writing poetry

One thing I had noticed right from the beginning is how much fun it is to write Lisp. I once told a friend: “Writing C++ is like writing a technical manual, Java like a school essay, Python like an email to a friend – but Lisp is like writing poetry.” I found it hard to explain at the time, but I think I may have an explanation now.

S-expressions

A lot of it has to do with Lisp's use of S-expressions. Basically, all Lisp code consists of nested expressions of the form (<function/keyword> <args>) For example, the following function checks whether a given list is a set (i.e. no elements occur more than once):

(defun set-p (lst)
    (cond ((null lst) T)
        ((member (car lst) (cdr lst)) NIL)
        (T (set-p (cdr lst)))))

Coming from a C-based world, that looks rather unusual. (What exactly each function call above does is irrelevant at the moment.) The important thing to realise is that every single S-expression has a value – and can thus be directly used as an argument to another expression. Unlike in imperative languages like C, one doesn't have to use as many one-time-variables. You can stick if-clauses and even loops straight into a function call. Instead of writing (in Java):

int a = arg;
if (a > cutoff) a = cutoff;
System.out.println(a);

you can simply write:

(format t "~&~S" (if (> arg cutoff) cutoff arg))

This style of writing code takes some getting used to, but in the end it makes for shorter and more elegant code.

Lastly, note that Lisp does not have any operators. +, -, <, >, etc, are all functions. Everything is an S-expression, giving a very fluent feel to the language.

Lists

LISP stands for “List processing”, and handling lists is what Lisp does best. Turns out, lists are incredibly versatile data structures. Though they may not always give the best performance, they are fantastically simple to use. Also, building other data structures on top of them (such as trees or graphs) is trivial.

Common Lisp provides plenty of functions to work with lists, and defining your own is easy. (Should you need the extra performance, Lisp does of course also offer arrays and other data structures.)

Macros

Macros are often touted as the single greatest advantage of Lisp. I have to say I still have not understood them completely (it's a complex topic), but they do allow you to do some pretty cool things.

Unlike C preprocessor macros, Lisp macros are an integral part of the language, with the whole expressive power of the language available to them. Basically, they can be used to extend the very syntax of the language.

There are many macros built in to Common Lisp. For example, there are at least five looping constructs availabe in standard CL. This may sound like an overkill, but it actually means that you can use precisely the right form for the job at hand – minimising lines of code and increasing readability.

Simple macros are often used to abstract away boilerplate code, for example when declaring numerous subclasses. More complicated macros (such as the built-in loop macro) can do wonderous things with the language … but I am not sufficiently well-versed in “macrology” to talk authoritatively on the subject.

Functional programming and object orientation

Historically, Lisp has been seen as a functional programming language. This is a style of programming that emphasises the role of, well, functions. Ideally, these functions should not have any side effects, like modifying global state (i.e. a function should always return the same output from the same input). This differentiates functions from procedures – the latter is a set of instructions, the former a discrete mathematical construct. It's too big a topic to expound here, but writing side-effect-free functions like this has some very tangible rewards, especially when it comes to debugging and testing a program. It is also a much more natural way to approach certain problems than the currently ubiquitous object-oriented paradigm.

Related to functional programming is applicative programming, which involves passing functions around as arguments. Lisp has (indeed, it invented) functions as first-class objects. This means that functions can be returned by, created by, and passed to other functions. Again, this lets you do some really neat things.

But Common Lisp is not only a functional programming language. It also offers a powerful facility for object-oriented programming, known as CLOS (Common Lisp Object System). This is actually a lot more flexible than the OO approach used by Java and similar languages, though it does take some getting used to.

In short, whatever style of programming you prefer, Common Lisp has you covered.

From prose to poetry

So why is Lisp such a joy to write? I think the short answer is that it is a language that gets out of your way. It lets you say exactly what you want to say. By default it already offers you a lot of different constructs to choose from – and if you don't like any of them, you just whip out a macro and create your own. You can always choose exactly the right tool for the job. It also encourages abstraction: programs that are built up in layers, with each successive layer not having to worry about the nitty-gritty of whatever is going on below it.

And because of these two things, you end up writing code that is terse and succinct, that gets a lot done in just a few lines. This is not only faster to write, but also easier to read – and therefore easier to conceptualise, extend and debug. Aside from the beauty of the balanced parens, I believe that this is the source of the aesthetic appeal of Lisp.

The State of the Union

Thus far on the language itself. But what about the state of its ecosystem? Programmers rarely create any sizeable program purely with the core language, but usually rely on third-party libraries. The following is a very brief overview of the Common Lisp world in 2018.

Implementations

As mentioned above, there are various implementations of the Common Lisp standard available. Each has their own strengths and uses. SBCL, the most popular open-source implementation, is known for its high-quality compiler. (Well-written code compiled with SBCL can reach near-C speeds.) GNU CLISP is not as fast, but more user-friendly. ABCL compiles to Java bytecode, ECL can be embedded in C applications. And the list goes on…

This range of options can be a little intimidating to a newcomer, but in most cases, the differences shouldn't matter (as each implementation conforms to the Common Lisp standard). If in doubt, use SBCL.

ASDF and Quicklisp

The most fundamental of all CL libraries is asdf – indeed, it comes pre-shipped with most modern implementations. ASDF (Another System Definition Facility) is a build system for Common Lisp projects. Comparable to make, it is used to define dependencies between different project components and on third-party libraries. It also stores metadata about the project. Though not part of the Common Lisp standard, it has become the de facto standard for distributing Lisp software.

Built on top of ASDF is Quicklisp, a package manager and repository for Common Lisp libraries. Quicklisp provides a simple user interface to load and install external libraries. Pretty much all important projects are available for download via Quicklisp – the repository currently holds over 1,500 libraries (and is updated monthly).

Available libraries

A comparatively small community means that Common Lisp does not have as rich a package ecosystem as, say, Python. Its standard library is also pretty small. (In Python-speak, it does not come batteries included.)

Nonetheless, though highly specialised packages are not to be expected, the Quicklisp repository does offer a good, broad selection – ranging from the likes of croatoan (an ncurses wrapper) to coleslaw (the static site generator this blog is built with).

I don't have a huge amount of experience yet, but from what I've seen, the average quality seems to be fairly good. There's a couple of “big-name” libraries around: alexandria for general utility functions, bordeaux-threads for multi-threading support, hunchentoot as a web server, etc. On the whole, documentation tends to be decent – ranging from “minimal but existent” to “very good”. Don't expect a whole lot of online tutorials, though. Despite this, I have had remarkably few problems working with the libraries I have used so far – due in large part to the fact that their source code is eminently readable. Most of them don't have a very large code base, but even so: a quick browse and some grepping is usually all that's needed to discover the specifics of a certain feature. I have found this a very pleasant experience; and vastly different to, say, trying to understand the internals of a PHP Nextcloud plugin.

The community

Common Lisp has a small, but capable and active community. There's not a lot of people who write Lisp; but those who do tend to know what they are doing. There's various IRC channels and mailinglists for those who are interested, as well as a handful of blogs. I've gone to the #lisp IRC channel for help before and found the people there friendly and helpful.

Personal conclusion

I thought I knew Common Lisp after my first project, three years ago – but now, even after another two months of intense learning, there's still a whole lot left to learn. Although the basics of the language are not hard to master, Common Lisp is a very large and complex system on the whole (unlike its much smaller sibling, Scheme). That makes it clunky and sometimes even ugly in places, but all the more exciting to explore :-)

On the whole, though, it is a beautiful language that is truly fun to work with. I wish I would have the chance of working in it at university. (At least we are using Julia in my research group – a language that has taken on almost every Lisp feature, short of S-expressions.) We'll see, I may have a chance yet. And until then, there'll always be side projects to work on…

Further reading

Addendum

Somebody posted this article on Reddit, whereupon I got an email comment pointing out two further resources that merit inclusion in the above list:

Also, in the list of implementations above, I should have mentioned Clozure (aka CCL), as it is another mature and popular implementation that also works on Mac and Windows.

Tagged as computers, programming, lisp, favourites

Unless otherwise credited all material Creative Commons License by Daniel Vedder.
Subscribe with RSS or Atom. Powered by c()λeslaw.