Why CLVM? What's the Problem?
Today Lisp implementors still spread their efforts over a lot of platforms developing a good code generation and platform-dependent optimizations ("good" here, in particular, means "run-time compilation to native code is supported"). Any transfer to any new platform is painful and might be done for every implementation with different success. Some CL-implementors try to resolve this issue in different ways.
For better understanding the point please visit a nice FAQ on Parrot, which answers similar questions but concerning VM for Perl. See also benchmark page, especially The Great Computer Language Shootout, where you can see that CMUCL (the one of the fastest CL implementations) is not that competitive.
- How do different implementations resolve the issue "compile to native"? Here are some examples:
- JVM is used in Bigloo and kawa (well, it is not a CL, it is Scheme.)
- ECL and GCL make this via C (and therefore they are not good in run-time compilation of one function to a native code)
- CLISP they just deny compilation to native ;)
If CLVM could have been created already and even have been used by CMUCL and SBCL then we could have those nice CL implementations under win32 today (i.e. July 2003) too.
How not to spread resources of Lisp implementors?
In the context of the problem given the answer is CLVM. And then CL implementors could focus their attention on CL front-end implementation itself.
Does CLVM exist today?
Strictly -- No.
-
There are interesting candidates however:
- RTL/Tree from GCC (Per Bothner and other say that RTL/Tree is too much internal and hardly good for compilation in run-time, but still it is not impossible)
- Parrot
- LLVM (sounds very promising, and version 1.0 is released in October 2003!)
- SEAM (Simple Extensible Abstract Machine) [PDF]
What does CLVM consist of?
- Virtual Machine (VM) itself
- CLVM-code
- JIT optimizing compiler for CLVM
Some comments on the terms of this formula see below.
VM
VM should be compliant for Lisp languages (and not just optimized for current platforms available and not just for current industrial languages like Java, C/C++ and a like)
As an example, one could learn how Bigloo guys use a JVM in order to make something better here.
Without a very good Lisp oriented virtual machine(s) we shouldn't think about a nice Lisp oriented hardware.
CLVM-code
For having idea what CLVM-code could mean, please look into byte-code of Java or in byte code of CLISP.
CL is too big to be fully mirrored into CLVM-code. There should be some kind of CL-kernel (or subset of CL) which is good for 1-to-1 corresponding with CLVM-code. One could even think of something like subset of Scheme.
JIT optimizing compiler for CLVM
Front-end CL-compilers should do their good job for translation of CL expressions into the optimized CLVM-code, and those optimization should nothing to have with platform-dependent optimizations. Here come JIT optimizing compiler for CLVM, they should optimize CLVM-code for given platform and run it using VM (see above).
CLVM Benefits
What brings us CLVM?
- easier port of CL-implementations based on CLVM
- better performance for CL programs (e.g., compilation to native, even in run-time)
- more free CL-developers
- CL-implementors could focus themselves on conforming their CL-implementations to a standard
- encouragement for Lisp hardware development
- quicker development of current and new Lisp implementations
What's the purpose of this page?
-
The purpose of this page is:
- to make the initial problem statement
- to debate solution ideas
- to attract positively-inclined CL developers to this problem
It is hardly possible to implement CLVM by two hands, therefore the main purpose of this page is to collect CL developers together who are motivated with the goal of implementing CLVM.
Sometimes it is a pity just to occupy the nice CL** name and it would be good if critical mass of CL-developers will appear around CLVM flag.
What's the audience for this page
- CL implementors (probably some of them could find it interesting)
- CL developers (perhaps, some of them could become implementors given the goals above)
- non-CL people (e.g. like Valery Khamenya, who is not able to switch from C++ to CL, because there is the only one CL implementation a bit close to typical good C++ environment like Kylix, but with price not for personal use)
- and a lot of other people
Feedback
Initial version is prepared by Valery Khamenya, feedback and other contributors are very welcome. Making comments straight forward on this page don't forget to put your signature or hint, please. Otherwise content becomes a bit unclear.Comments from Community
Some assumptions here seem suspect:
- Do CLISP people really object to a JIT backend? CLISP seems happy in its niche as it is, sure, but that doesn't imply the existence of actual objections to improvements, now does it?
- The intended audience is not clearly defined. is it the free CL implementors? those don't need a VM, for the following reasons:
- JIT'ed VM as a way to get performance is obviously not an improvement on CMUCL/SBCL. Actually, whatever the merits of this entire proposal, I don't agree with this statement. You can only get "optimal" performance in CMUCL/SBCL, or indeed in any Common Lisp implementation today, with a lot of intrusive declarations. There are many parts of the language, especially with respect to CLOS, that would benefit from a profiling + JIT approach that would mesh nicely with a VM. Tim Moore
- JIT'ed VM as a way to conserve effort does not seem to be needed. well, perhaps on the PPC where there is actually more than one free native-compiling CL implementation. there's just not enough effort duplication to justify the need for a common backend platform, IMHO.
- VM as a vehicle for non-free add-ons might be interesting, but doesn't seem a priority.
Comment by Faré: as far as I understand it, Common Lisp itself is the Virtual Machine, and SEX is its standard bytecode (well, charcode) representation. If you want a trimmed-down and/or extended VM on top of which to "implement" CL in a layered way, then it amounts to defining a Kernel Lisp (see Lisp - Next Generation for a discussion) -- and don't expect anything like a standardization until you have tremendously successful code. On the other hand, if what you propose is refactoring some existing Lisp implementation to add some portability features to it, well, go ahead! PS: you may be interested in reading the archives of the LispVM and related mailing-lists LispOS and virtmach. Good luck!
Comment by rpg (not that rpg!): I have been wondering about the possibility of building CL on top of the Parrot virtual machine for perl. That virtual machine (www.parrot.org) seems to have many of the features that would suit it to lisp-like languages. Question, would it be appropriate to piggyback on that work? I'm not suggesting this so much as trying to educate myself....
There is a beginnings of a lisp implementation on parrot at sprocket. The announcement is at http://www.nntp.perl.org/group/perl.perl6.internals/29074
- lapax
-
An answer from Valery Khamenya:
- CMUCL, SBCL and others are still not ported to Win32?
- Why some natively generated code is still a bit slower then in C?
If it is not worth of doing that, then why
I have thought about that a bit, I've even have a Parrot/languages/CL which reads/writes Parrot's byte code and can handle the most basic CL special ops. But the fact is: 1) writing a compiler is only fun at the start, 2) CMUCL, SBCL, OpenMCL, CLISP, Corman Lisp, Lispworks, Allegro CL, MCL 3) Writing a useable CL compiler is a LOT of work 4) in all honesty I'd rather be using Common Lisp than implementing it. -- Marco Baringer
Marco: That seems like an eminently reasonable position. Indeed, I agree with you --- I'd rather use CL than develop an implementation, too. That's why I studied AI instead of programming languages or compilers! I'm just trying to educate myself a little. -- rpg (not that rpg!)
As an SBCL implementor, I hack on free Lisp compilers because (a) I enjoy it, and (b) I hope that other people will find a free Lisp system useful and choose it for their own projects, and then decide to feed back some of their work into the free software community. Given that I don't enjoy using Windows, and it isn't free (so a Lisp system which requires it won't be free either), I'm not about to start spending my spare time on it. If you can produce a CLVM which works sufficiently identically on Windows and Unix that SBCL will run on both platforms without my even needing to install Windows to test it, I recommend that you stop talking about it and start coding. -- Daniel Barlow
I would like to comment on "there is the only one CL-implementation a bit close to typical good C++ environment like Kylix, but with price not for personal use". Franz, LispWorks and Digitool all offer "for personal use" or demo copies completely for free, while Kylix Professional costs USD 250. If you want to actually buy the product you'd have to compare not to Kylix Professional but to Kylix Enterprise which costs USD 1999, much more than the prices listed on Lispworks or Digitool for similar products. While I'm at it, I'd like to mention that I'll take SBCL or OpenMCL + ILISP over Kylix any day of the week. -- Marco Baringer (who is probably just feeding the trolls...)
Of course, this problem plagues other languages than Common Lisp, too. Sooner or later, users of any programming language are going to want the implementation to be more efficient. And sooner or later, the implementors are going to run into the issue of there not being a light-weight native code generation backend that works across all platforms they want to support and generates efficient code for all features of the language. Many language implementations roll their own code generator, which is then specific to that language, and usually hard to separate from that implementation. Some independent virtual machines exist, but most are not exactly lightweight, and they often make assumptions about language features. Unsatisfied with what I have found, I have decided to start down a different path. I am (in my copious free time) developing TurboVM. The driving force behind TurboVM is keeping it small and simple. It implements a simple RISC instruction set, which should make it suitable as a target for any programming language. Currently, TurboVM bytecode can be interpreted or compiled to C code. It is very fast in both modes (once the C code is compiled to native code, its performance is on par with an equivalent program written in straight C). Eventually, the goal is to support run-time native code generation without C as an intermediate step. -- Robbert Haarman
If I had free time, I'd be happy to write my own CLVM: it would help me learn Common Lisp better as well, and I'm especially attractive to the idea of tapping into JIT compiling, to make CL more efficient without having to resort to type declarations. It would also be especially nice if all we had to do to port Common Lisp would be to port a VM. I am currently working full-time, though, and am also looking for additional part-time work...now, if only someone could pay me to work on such a beast! -- Alpheus Madsen