Compiled Code Compiles Code

Now and again I try and do something that stretches the limit of a programming language. This usually results in a very bad headache and days of frustration.

At work I have been developing a rule based document annotation system called Saxon. The rules are loaded into the system and converted to Java code which is then compiled, loaded into the running instance of Java and then used to annotate the documents. If you are already getting a headache I'd stop reading here as this post is only going to get more confusing!

For a long time there was no standard way of having a running Java program compile and then use new classes. The only way was to write the source files to disk, call the command line compiler and then create a new class loader to load the compiled classes into memory. Not a simple way of doing things.

With Java 6 Sun introduced a standard API for compiling that was supposed to make life easy. Once you know how the new API works it is in fact very easy to use. On the plus side you can now easily do in-memory compilation which is much faster than reading and writing to disk, but of course you are restricted to using Java 6 which not everyone has upgraded to yet.

Unfortunately there is a much more serious problem. Sun, in their infinite wisdom, decided that compiling code was something only developers would want to do and so only bundle a concrete implementation with the JDK and not the JRE which means that applets and Web Start applications don't have access to a compiler.

Fortunately Eclipse provide the JDT Core Batch Compiler which includes a Java 6 complient compiler. This works great in standalone apps, allowing me to run Saxon from a JRE and still have all the functionallity I need.

However, I was trying to make Saxon available as a Web Start application and ran into a bit of a problem. There is something very strange with the class loading in the Eclipse compiler. It works fine in a standalone application but under Web Start the compiler can only see core Java classes which means I can't reference my own API from the classes I'm trying to compile and everything fails and I end up back at square one.

Don't despair though as their is a solution. Throw away the Java 6 API and use the Eclipse compiler directly. This has the advantage of working under Web Start and under previous Java versions (at least version 5 which is more commonly installed than 6). I can't take all the credit for this solution but I thought it worth blogging about as I'm yet to see anything similar any where else on the web, although I did find lots of posts asking how to do this.

The following code is cobbled together from two main sources; Apache Tomcat and GATE. I actually used the GATE code, which is based on Tomcat code, as my starting point and then simplified it and added some generics support.

You can download the full compiler (a single Java source file and the JDT Batch Compiler) from here. And here is a simple example of how to use it (this is actually the main method of the class for easy testing).

EclipseCompiler compiler = new EclipseCompiler();

String HELLO_WORLD_CLASS_NAME = "HelloWorldTest";
String HELLO_WORLD_SOURCE = "import org.eclipse.jdt.internal.compiler.Compiler;\n\n" +
"public class "+HELLO_WORLD_CLASS_NAME+"{\n" +
"    public static void main(String[] args) {\n" +
"        System.out.println(\"Hello world\");\n" +
"    }\n" +
"}";

Class compiledClass = compiler.compile(HELLO_WORLD_CLASS_NAME, HELLO_WORLD_SOURCE);

if (compiledClass==null)
{
    System.out.println("Unable to compile file");
    return;
}

Method m = compiledClass.getMethod("main",String[].class);
m.invoke(null, new Object[]{null});

The interesting thing to note is that the small test case imports a class not from the core Java APIs and this works in standalone applications and when run under Web Start.