Skip to content

Babel Syndrome

At this moment the groups that will bring us the seventh revision of the report on the algorithmic language Scheme (R7RS), which is going to be split in two, are being formed. I just hope that they keep in mind that even the gods will fear us if they succeed at producing a report (or reports) that the vast majority of us like:

“If now, while they are one people, all speaking the same language, they have started to do this, nothing will later stop them from doing whatever they propose to do.” – Genesis XI, v.6

Scheme hits the App Store

Reverso logo

I believe I have got the first Scheme application past Apple review into the iTunes App Store. It is yet another Reversi clone, called Reverso. It is a combination of 90% Scheme and 10% Objective-C, written with Gambit-C Scheme. James Long has already shown how to compile Gambit-C for the iPhone, and I started from there. My Scheme code is compiled to C by Gambit and later by GCC to produce native ARM code, bundled in a static library, which is ok with the iPhone SDK license agreement. The Objective-C then calls the library as a pure C library. The Scheme code deals with position evaluation, alpha-beta pruning, transposition tables, move legality, different strategies and so on. The Objective-C code deals with sound, animations, GUI, user preferences, basically everything that calls the iPhone OS API. Reversi was chosen because I like strategy games and it is much more algorithmic than artistic, and I am no artist.

The performance of the code is excellent. I used some Gambit-specific declarations, only fixnum arithmetic, pre-allocated a large heap, and called the garbage-collector every time the user needed to think. The search is not memory intensive, but the transposition tables are. I did not write a specific hash function but relied on Gambit-C’s table type. The boards used during search were retrieved from a pool (the newest Gambit-C has made subu8vector-move! a public API), so they did not put pressure on the garbage-collector. In the end it was a very successful experiment. Developing with Scheme is orders of magnitude more productive than with most other languages. Gambit-C is also one of the best Scheme compilers out there, and made my job a lot easier.

Update: There was a bug in the application that caused it to play poorly. The weak AI was not Gambit’s or Scheme’s fault, but mine. I already sent an updated version to Apple that will play much better. Besides fixing the bug, I am now using Zobrist’s hashing instead of Gambit’s own hash function, that performs poorly for game positions. It is still written in Scheme, though. Now I am waiting for the approval of the update, and to hear further feedback from my users. :)

Sly Scheme

“You think you know when you can learn, are more sure when you can write, even more when you can teach, but certain when you can program.” – Alan Perlis

Just after have read Lisp in Small Pieces I felt the urge to write a Scheme compiler. This sentiment is common in the Lisp community, I guess. There are dozens of Scheme systems, some good, some bad, some maintained, some abandoned. There are large optimising compilers and small interpreters for embedded systems. But I, well, needed to know. I needed to know how to write a program that takes another as input and generates simple instructions that do the same computation. I needed to know how to a write a program that interprets those instructions, a runtime to provide primitives, a garbage collector to manage memory etc. So I started doing some coding in my spare time, and called the monster Sly Scheme.

I failed miserably. I began with the reader, believing that basic I/O was fundamental for a REPL (the REPL was the goal I set myself to achieve). The reader was growing too complex and my interest simply disappeared. Then I focused on the types, creating a large hierarchy of them, only to lose interest again. After some time I came across Abdulaziz Ghuloum’s An Incremental Approach to Compiler Construction, which starts with the compiler itself. Let another Scheme do the I/O! I then took this path, running my compiler with Gambit-C. But instead of generating machine code, it generated instructions for a virtual machine. I then proceeded in lockstep: Wrote in Scheme a new compiler feature, and then in C just enough to interpret the new instructions. Finally most of R4RS Scheme is compilable. Then I wrote a simple stop-and-copy garbage collector with two semi spaces. After that, I needed a runtime with enough procedures to run the compiler without Gambit-C. Almost an year later, I have my REPL (I am not that bad as a programmer, but this is just a hobby that I do in my spare time).

The code now sits in a Github repository. For those who cares to check it out, I must warn that everything was done as simplistic as possible, so I could have a working interpreter ASAP. Some things were even done outright stupidly. The compiler generates too many closures, it fails with simple cases, the VM uses a giant switch, it is slow, the garbage collector too etc. I probably will hack this code for the years to come, even if only as a hobby of mine. But the most important contribution of this project is how much I have been learning from it.

Escape from Zurg

Lately I have been reading about searching in game trees. So a friend of mine sent me the “Escape from Zurg” paper, which talks about problem solving by tree searching with Haskell.

In a large class of problems one is given a start state and some predicate for a desired final state. Moreover, the rules that dictate how a successor state is generated from a previous one are also given. Take a board game for instance, like tic-tac-toe. There is the initial state (which is the empty board), the predicate for a desired state (a state in which tree pieces of mine are aligned), and rules to create one state from another (the rules of the game). Another example of such problems is the “Escape from Zurg” one, that is stated as follows:

Buzz, Woody, Rex, and Hamm have to escape from Zurg. They merely have to cross one last bridge before they are free. However, the bridge is fragile and can hold at most two of them at the same time. Moreover, to cross the bridge a flashlight is needed to avoid traps and broken parts. The problem is that our friends have only one flashlight with one battery that lasts for only 60 minutes (this is not a typo: sixty). The toys need different times to cross the bridge (in either direction):

TOY TIME
Buzz 5 minutes
Woody 10 minutes
Rex 20 minutes
Hamm 25 minutes

Since there can be only two toys on the bridge at the same time, they cannot cross the bridge all at once. Since they need the flashlight to cross the bridge, whenever two have crossed the bridge, somebody has to go back and bring the flashlight to those toys on the other side that still have to cross the bridge. The problem now is: In which order can the four toys cross the bridge in time (that is, in 60 minutes) to be saved from Zurg?

This class of problems is usually solved by searching. The initial state and all the successors ultimately build a DAG (directed acyclic graph) that is called the search space. Some search spaces are small, like tic-tac-toe playing or even the “Escape from Zurg” puzzle. Others are large, like playing Reversi; even larger, like playing Chess; others are infinite. Needless to say that that are better searching ways than others. This same puzzle, for instance, was solved in Scheme by Ben Simon using the cool amb operator. The amb operator uses first-class continuations to backtrack and take another path in case the first one fails. It was an elegant solution for this puzzle, but it is not adequate for general searching because it backtracks blindly. There are others ways to search blindly, for instance depth-first search and breadth-first search.

In a depth-first search, we analyse the successors of a given state until there are no more successors to analyse, and then backtrack to a previous level of the tree. In a breadth-first search, we analyse the whole fringe of the tree (the horizon nodes) before expanding it one level further using the successor rules. This is good for infinite search spaces, because we will eventually find a solution if there is one. The implementation in Scheme is straightforward:

 
(define (depth-first states successors goal?)
  (if (null? states)
      #f
      (let ((state (car states)))
        (if (goal? state)
            state
            (depth-first (append (successors state)
                                 (cdr states))
                         successors
                         goal?)))))
 
(define (breadth-first states successors goal?)
  (if (null? states)
      #f
      (let ((state (car states)))
        (if (goal? state)
            state            
            (breadth-first (append (cdr states)
                                   (successors state))
                           successors
                           goal?)))))

This is actually all we need for simple searches. Of course we need to supply the initial state, the procedure that generates successors and the goal predicate. These are specific for each kind of problem. Let’s see the Zurg problem procedures:

 
;; the type of the states found in our puzzle
(define-type state
  (previous unprintable:)       ;; previous state
  near                          ;; toys in near shore
  far                           ;; toys in far shore
  flashlight                    ;; location of flashlight
  time)                         ;; time consumed so far
 
(define (start-state)
  (make-state #f
              '(buzz woody rex hamm)
              '()
              'near
              0))
 
(define (final-state? state)
  (and (null? (state-near state))
       (<= (state-time state) 60)))
 
;; at most two toys can travel across the bridge
;; the flashlight must be used in all crossings
(define (successors state)
  (let ((orig (if (eqv? (state-flashlight state)
                        'near)
                  (state-near state)
                  (state-far state)))
        (dest (if (eqv? (state-flashlight state)
                        'near)
                  (state-far state)
                  (state-near state))))
    (map (lambda (toy-pair)
           (let ((toy1 (car toy-pair))
                 (toy2 (cdr toy-pair)))
             (make-next-state state
                              (remove toy2
                                      (remove toy1 orig))
                              (ucons toy1
                                     (cons toy2 dest))
                              (max (time-cost toy1)
                                   (time-cost toy2)))))
         (product orig orig))))

The previous procedures use some utilities:

 
(define (make-next-state state orig dest time)
  (if (eqv? (state-flashlight state) 'near)
      (make-state state orig dest 'far (+ (state-time state) time))
      (make-state state dest orig 'near (+ (state-time state) time))))
 
;; the time each toy takes to cross the bridge
(define time-cost
  (let ((costs '((buzz . 5)
                 (woody . 10)
                 (rex . 20)
                 (hamm . 25))))
    (lambda (toy)
      (let ((pair (assv toy costs)))
        (and pair
             (cdr pair))))))
 
(define (ucons a b)
  (if (memv a b)
      b
      (cons a b)))
 
(define (product lis1 lis2)
  (let lp1 ((lis1 lis1)
            (res '()))
    (if (null? lis1)
        res
        (let ((i1 (car lis1)))
          (let lp2 ((lis2 lis2)
                    (res res))
            (if (null? lis2)
                (lp1 (cdr lis1) res)
                (let ((i2 (car lis2)))
                  (if (member (cons i2 i1) res)
                      (lp2 (cdr lis2) res)
                      (lp2 (cdr lis2) (cons (cons i1 i2) res))))))))))

Our successors procedure does not take time in account. So it is possible for a branch of the tree to go on indefinitely. To guarantee that we will arrive at a solution, we use a breadth-first search:

 
(define (print-path state)
  (let loop ((state state)
             (path '()))
    (if state
        (loop (state-previous state)
              (cons state path))
        (for-each println path))))
 
(print-path (breadth-first (list (start-state)) successors final-state?))

With the result:

#<state #14 near: (buzz woody rex hamm) far: () flashlight: near time: 0>
#<state #15 near: (rex hamm) far: (buzz woody) flashlight: far time: 10>
#<state #16 near: (woody rex hamm) far: (buzz) flashlight: near time: 20>

#<state #17 near: (woody) far: (rex hamm buzz) flashlight: far time: 45>
#<state #18 near: (buzz woody) far: (rex hamm) flashlight: near time: 50>
#<state #19 near: () far: (buzz woody rex hamm) flashlight: far time: 60>

If we change our successors procedure a little, we can use depth-first search:

 
(define (successors2 state)
  (let ((orig (if (eqv? (state-flashlight state)
                        'near)
                  (state-near state)
                  (state-far state)))
        (dest (if (eqv? (state-flashlight state)
                        'near)
                  (state-far state)
                  (state-near state))))
    (filter (lambda (s)
              (<= (state-time s) 60))          ;; filtering out bad states
            (map (lambda (toy-pair)
                   (let ((toy1 (car toy-pair))
                         (toy2 (cdr toy-pair)))
                     (make-next-state state
                                      (remove toy2
                                              (remove toy1 orig))
                                      (ucons toy1
                                             (cons toy2 dest))
                                      (max (time-cost toy1)
                                           (time-cost toy2)))))
                 (product orig orig)))))
 
(print-path (depth-first (list (start-state)) successors2 final-state?))

The advantage of changing the successors procedure is that depth-first searches are usually faster than breadth-first searches, in this case 483ms vs. 24630ms. Also, it uses less memory. So it is always a good idea to try to cut the search space as soon as possible.

Is the App Store the way?

Last year I chose some goals for my career:

  • Work with very clever people or alone, from home
  • Work with good development tools, i.e., Git, Mercurial, Emacs etc.
  • Work with good programming languages, i.e., Scheme (first choice), Haskell, Common Lisp or Lua

These goals are intended to guarantee my long term happiness and stress-free high productivity. I mean a rewarding professional life. So far I could not attain any of them. Here in Brazil the chance of getting a regular job with these traits is nil. I began then to look for freelancing work, but I could only find worthless Java/PHP/ASP/VB/etc. projects. Unfortunately, here, those are the “high-tech” jobs.

As seen in my previous post, it is possible to use Scheme to develop for the iPhone and for the iPod Touch. Developing for the App Store has the potential of fulfilling my goals. Of course it is an already crowded market, but so far it seems to be the only way out. So I paid Apple and registered as an iPhone developer. Let’s see what I can come up with.

Writing apps for the iPhone in Scheme

James Long wrote a nice and comprehensive article about using Gambit-C Scheme for the development of iPhone applications.

Adding a garbage collector

I guess it is clear by now that I am writing a toy Scheme compiler and virtual machine. Primarily for learning the techniques, I just touch it from time to time. The last addition to the virtual machine was a Cheney-style copy garbage collector. I wanted to implement a very simple algorithm to get a functioning system quicker. The simplest algorithms I know of are the Cheney one and the Deutsch-Schorr-Waite mark-and-sweep garbage collector. I chose the copying one because I believe generational garbage collection is the way to go for Scheme virtual machines, given that the rate of allocation is very high (allocation in a copy garbage collector is just moving a pointer) and most data objects die young (how many times did you use reverse after a loop?).

I was surprised by how easy it was to implement the collector. It is a very simple one (stop-the-world, two semi-spaces etc.) but nevertheless I always thought garbage collectors were somewhat magical, a bunch of elves that worked out of sight to keep your process working perfectly. This is exactly why I am writing such a system, even if it never gets free onto the world it will fulfill its purpose. Of course, if I am satisfied with the result, I will release the beast.

My voter registration

The voter registration period for the Scheme Steering Comittee is over. I thought I could then post my statement of interest:

I have been using Scheme for almost four years now. I use it mostly for studying programming languages and programming in general, prototyping and testing new ideas, but I have also developed and sold applications written in Scheme to customers. I believe Scheme is nowadays a maximum in the design space of programming languages and is the language of choice of some of the best hackers of the world. I thus deeply wish that Scheme gets getter with each new standard revision.

Compiling let

Following my previous Scheme toy interpreter in Lua, I have already tried twice to write a simple Scheme compiler. I failed mostly because the complexity grew too fast and I could not see any of it working before losing interest. But I then read Ghuloum’s paper on incremental compiler construction. It’s a very nice approach, and I stopped trying to think everything in advance and instead started to work little by little. Unlike the paper I’m not compiling to assembly but to a virtual machine, tough.

I have come across one of the many design decisions a compiler writer faces. When compiling let forms, they are first translated to the closed lambda applications they are syntatic sugar for. For instance:

(let ((a "Alex")
      (e (+ (* pi 2) 1))
      (i (car (cons 1 0))))
  ((if a + -) e i))

Becomes:

((lambda (a e i) ((if a + -) e i))
 "Alex" (+ (* pi 2) 1) (car (cons 1 0)))

There is no need to emit code to create full closures when compiling closed lambda applications, because they cannot be applied multiple times. Instead, the environment is extended during compilation and the body is compiled in this new environment, but still it is considered just a piece of the enclosing body. So far so good, but what does happen during execution?

During the execution of the virtual machine, the arguments to full closures are pushed onto the stack (along with a return address, saved frame pointer and number of arguments pushed), before jumping to the closure code. Currently, when entering the body of closed lambda applications, a lightweight closure calling protocol is used: Just the frame pointer and the arguments are pushed onto the stack and are latter popped. But when accessing bindings not local to the current scope, which is very common in let forms when one access the arguments of the enclosing full closure, the compiler has to emit code for jumping through activation frames until the desired one is found, and the binding is found there. In other words, the VM instruction for accessing bindings is LOAD i j, where i is the activation frame level and j is the index of the binding inside that activation frame.

Although this works, it is not as fast as it could be. I am planning to use flat environments as described in Dybvig’s PhD thesis so all access to bindings inside closures will have constant time, not linear on the depth of the desired binding. The current compilation of closed lambda applications defeats that optimisation, although it makes bindings somewhat shallower. The alternative is to look for all the bindings introduced by closed lambda applications during compilation and always allocate space for all of them in the stack. This uses (possibly much) more memory because allocates space for bindings that may never be needed during execution, but accessing any binding is again constant time because all of them are known at compile time.

As usual, it boils down to memory versus speed. I guess I’ll leave it as is for now and worry more about speed after the compiler is able of even compiling itself.

Blog under siege

I know it is a pain for who is commenting, but I had to add the reCAPTCHA Wordpress plug-in to this blog. The spam count was already in the 300s, and increasing fast. Trackback spam is still high.