Today I found an interesting post about getting work using non-mainstream languages.

Well, unlike Java jobs or C# jobs or VB jobs, where one can scan the newspaper or go to dice.com, these magical jobs are not the ones I get when recruiters solicit me (three times a day). No, they appear between the cracks of the sky when it rends in two. All but one contract (that did come from dice.com) came from the company CEO pulling me aside or phoning me out of the blue from the materials I had published inside the company where I was employed, or from my web-sites on programming languages.

This basically means you must be a reference. You must blog a lot or, better yet, write even a book. When your name is said in a company meeting, somebody of the team must already have heard of you. Needless to say, this is not easy at all.

Usually and because of the exclusivity of the work it takes anywhere from at least 3 months to 2 years to secure these kinds of contracts, so the adage “Don’t quit your day job” is an appropriate one here. When I do get these contracts, however, they usually last longer (3 years) than the Java/C++/XML/web-services-code-grinder ones (that usually last around 6 months), and the peer group is much more intelligent, genteel and just plain more interesting than the code-grinding crowd. Is is harder to find these magical contracts? Yes, they are rarified air. But are they worth preparing for and then finding? Definitely.

Here we see that even after all this work, it is still very hard to find these dream jobs. You can wait as long as two years to get one, simply because in most companies the management employ Java/C#/VB drones anyway, as they are cheaper and easier to find. Only when they face really hard problems they start thinking of alternatives.

This post only reinforces what was already in the back of my mind for a while: The least hard way to earn money and be happy at the same time is to start an own business. Now if I could only move my lazy ass…

I have come across an application of XML to store an animation, frame by frame, like the animated GIFs of old. Each frame is then an image, stored as an XML element. The bizarre part is that every pixel is stored as an individual element inside the frame:

<frame id="Frame001" width="180" height="240">
  <array type="Int8">
    <Int8 value="230"/>
    <Int8 value="012"/>
    .
    .
    .
  </array>
</frame>

I believe this is the most “by the book” implementation. But this is far from optimal. When inspecting the files with a text editor (yes, I needed to do that) it was very painful. Besides, the files are huge! Why not be sensible and use Base64 encoding for the image part:

<frame id="Frame001" width="180" height="240">
  TWFuIGlzIGRpc3Rpbmd1aXNoZWQsIG5vdCBvbmx5IGJ5IGhpcyByZWFzb24sIGJ1dCBieSB0aGlz
  IHNpbmd1bGFyIHBhc3Npb24gZnJvbSBvdGhlciBhbmltYWxzLCB3aGljaCBpcyBhIGx1c3Qgb2Yg
  dGhlIG1pbmQsIHRoYXQgYnkgYSBwZXJzZXZlcmFuY2Ugb2YgZGVsaWdodCBpbiB0aGUgY29udGlu
  dWVkIGFuZCBpbmRlZmF0aWdhYmxlIGdlbmVyYXRpb24gb2Yga25vd2xlZGdlLCBleGNlZWRzIHRo
  .
  .
  .
</frame>

Nobody could inspect the information of previous format anyway, the files gets much smaller, one could edit them by hand (like I did to copy frames between animations). And no, the files are not being compressed because they are part of a build process.

It only remains to be seen why it is done this way. I believe it was a mixture of a non-too-intimate knowledge of the technology and a desire to use it in it “purest” form, like those people who must design their programs with all the design patterns known to man. With time they may learn that a sensible dose of pragmatism is very healthy.

• Essentials of Programming Languages, by Daniel P. Friedman, Mitchell Wand and Christopher T. Haynes
• Programming Language Pragmatics, by Michael L. Scott
• Smalltalk-80: The Language and Its Implementation, by Adele Goldberg and David Robson
• Writing Compilers and Interpreters: An Applied Approach Using C++, by Ronald L. Mak
• Modern Compiler Implementation in ML, by Andrew W. Appel
• Lisp in Small Pieces, by Christian Queinnec
• An Incremental Approach to Compiler Construction, by Abdulaziz Ghuloum
• Let’s Build a Compiler, by Jack Crenshaw
• Structure and Interpretation of Computer Programs, by Harold Abelson and Gerald Jay Sussman, with Julie Sussman
• Advanced Compiler Design and Implementation, by Steven Muchnick
• The 90 Minute Scheme to C compiler, by Marc Feeley
• Compilers: Principles, Techniques, and Tools, by Aho, Lam, Sethi and Ulman (the “Dragon Book”)
• Engineering a Compiler, by Keith Cooper and Linda Torczon
• Mike Pall’s guide to the Lua source code
• Anton Ertl’s papers
• David Gregg’s papers
• Michael Franz’s papers
• Garbage Collection: Algorithms for Automatic Dynamic Memory Management, by Richard Jones and Rafael D Lins
• Paradigms of Artificial Intelligence Programming: Case Studies in Common Lisp, by Peter Norvig
• The Art of the Interpreter of, the Modularity Complex (Parts Zero, One, and Two), by Guy Lewis Steele, Jr. and Gerald Jay Sussman
• Compiler section on readscheme.org

And anyway, the subject line presupposes that Lisp has not caught on. This is like saying that astrophysics or calculus or brain surgery has not caught on because in relative numbers, there might be more people doing other things. The success of Lisp is not measured in the number of people using it, it’s measured in the utility to those people who do use it. Turning it into C (or C++ or C#) to make it more popular would not be success. In the world’s menu of computer language options, we don’t need them all to be Taco Bell.

I have departed from the traditional approach of implementing a Scheme interpreter in Scheme itself because I wanted to avoid possible confusions between the defined language and the defining language. This has made a lot of the concepts clearer. The evaluator is written in continuation-passing style, which is easily done in Lua because the latter has tail-call optimisations. This way it is easier to reify continuations to first-class values.

I have also added a few primitives to allow me to run some examples, as the factorial and fibonacci functions. More primitives can be easily added if desired.

Ok, I know the World is already sick of Schemes, it only takes a look at this list to know why. But I still think I have to do it, if only for my personal learning. There is no need to bore the World to death with another half-done Scheme. Sometimes I think about features that could make my Scheme useful, though. They are probably never going to be implemented, but here they are:

Full R5RS compliance

The R5RS is a piece of art (R6RS, on the other hand, is a stain, a mess). Of course it does not describe a terribly useful system, but it beautifully describes the core of one. Everything else can then come from there. For more compliance with other systems, the SRFIs can be used as the basis for a standard library.

Bytecode virtual machine

Although Scheme is a dynamic language, there is a lot that can be known about the program before executing it. Lexical variables, never assigned variables, closure and continuation creation etc. This can be processed at compile time, allowing for a simple and fast virtual machine. Interpreting the source code directly is easier but very inefficient for any serious system.

Incremental garbage collection

Stop-and-go garbage collectors are easier to implement, but in some applications the pause is inacceptable. For applications like interactive games, the smooth user experience only can be achieved by an incremental garbage collector. It would be nice if it was a moving collector too, to avoid too much fragmentation of the heap (some naive C programmers think the use of malloc and free can beat a garbage collector in any circustances, until they are hit in the face by heap fragmentation. There is a reason the Apache web server uses memory pools). If I remember correctly the CLR garbage collector is very good and compacts the memory without the use of from- and to-spaces, which is cool, because uses less heap memory. Hmm, while I am dreaming about my ideal garbage collector, make it concurrent too (see below).

Concurrency

These days the hot topic is concurrency. New personal computers with two and four cores sell more and more every day. The programming languages/environments that makes easy for the programmer to use these new cores are going to be big. Erlang is the hottest thing right now in this regard, although Haskell is a strong candidate too. In the mainstream Java has a concurrent VM, but Java uses the old lock/unlock paradigm of other mainstream languages, Clojure is the new language on top of the JVM that leverage that power for the programmer. But although this is all so hot, the only thing that seems to be happening to Scheme is Termite, which as far as I know can only use green threads. My hypotetical Scheme would have a concurrent VM, some Scheme compilation techniques already give a start in that direction. There would be maybe a spawn-continuation form that would take a continuation captured with call/cc and resume it in parallel with the current one, in a M:N model (M continuations spread on N OS threads).

Just-in-time compilation

Well, once dreaming, why not go all the way? One of the nice things about Scheme is the bottom-up, incremental development. Developing at the REPL gives instant feedback and ease debugging. What if it could give the speed of compiled-to-CPU languages as well? Another advantage is hot-swapping code in a running application. This can’t be done with Chicken or Gambit, which are fast, but Scheme-to-C compilers. Larceny, Chez and Ikarus can do this. By using a bytecode VM, the system is portable to wherever there is a C compiler, and by adding JIT support for some architectures it can be made fast too.

Escape continuations

Full, first-class continuations with indefinite extent are a powerful and mind-bending feature, but unfortunately, are usually hard to implement and causes pauses when invoked. Besides them, my hypotetical Scheme would have escape continuations. Instead of completely overwriting the execution stack, they simply unwind it until some recorded point, which can be made very fast. Exceptions would them be implemented as escape continuations rather than full ones.

Easy FFI

I am a long-time Lua user, and it is amazing how they could get the C API so good. Not only it is very easy to add new C/C++ libraries to Lua, it is trivial to embed the Lua interpreter itself in a legacy application, or an application that needs to be mostly in C/C++, such as games. As Scheme, unfortunately, is not one of the World’s favorite languages, the capacity to talk to the outside World is fundamental. And by being embeddable (which is the goal of Guile, but it sucks for Windows), it would be easier to take Scheme to another domains where it is not so popular today.

Well, I guess this is it. Thinking about it, there seems to be no Scheme system today with all these features. Will I have the skills to write something like this after LiSP? I doubt it, only the garbage collector would take me ages. But it would be nice if such thing existed…

Although the focus of Dybvig’s thesis is not on concurrency, I cannot help but think that his compiler could work very well for this too with just minor modifications. If variables (local or free) are never assigned, they could be replaced by their values and several different threads with different execution stacks could run in parallel. If a variable is assigned, the access to its indirection box could be protected by a mutex or similar synchronisation primitive. Of course this would make access to assigned variables even slower, because even variables ever used by only one single thread would need mutex lock/unlock operations. On the other hand, isn’t assignment discouraged in Scheme? The overall execution time would compensate for this, if the rest of the program was parallelised.

But surely I am missing something, because no one ever seems to think Scheme can be a contender in the massive-parallelised future. What bugs me is that I don’t know what it is that I am not taking in consideration.

Unfortunately at work I am stuck with Windows. I am a long-time Vim user so I have it installed, which adds an “Open with vim…” entry to the Explorer shell. This opens a new Vim instance everytime it’s used. A similar approach won’t tempt Emacs users because you do not open Emacs everytime you open a file. You open Emacs once a week or once a month and keep it running. So enter emacsclient.

It acts as an independent program for an external caller. When started, this program looks for an already running instance of Emacs and connects to it, which then acts as a server and edits the requested file (to accept connections, server-start must have been run in Emacs. I have it in my .emacs file). When the user is done, he types C-x # in the server buffer. Then emacsclient exits and the external caller assumes the editing application has quit.

This is very useful, for instance, for writing commit log messages. My EDITOR environment variable points to emacsclient. Everytime I need to commit code in Subversion or Monotone, the log message appears as a new buffer in my running Emacs. It is also useful for the Firefox extension It’s All Text!, which allows one to use an external text editor to edit web forms. And, of course, I have added an “Edit with Emacs…” entry to my Explorer shell, so I can open any file into a running Emacs very quickly.

Instructions about how to add the shell shortcut can be found in the Emacs wiki.