Thursday, 16 October 2008

Compiled Code Compiles Code

Now and again I try and do something that stretches the limit of a programming language. This usually results in a very bad headache and days of frustration.

At work I have been developing a rule based document annotation system called Saxon. The rules are loaded into the system and converted to Java code which is then compiled, loaded into the running instance of Java and then used to annotate the documents. If you are already getting a headache I'd stop reading here as this post is only going to get more confusing!

For a long time there was no standard way of having a running Java program compile and then use new classes. The only way was to write the source files to disk, call the command line compiler and then create a new class loader to load the compiled classes into memory. Not a simple way of doing things.

With Java 6 Sun introduced a standard API for compiling that was supposed to make life easy. Once you know how the new API works it is in fact very easy to use. On the plus side you can now easily do in-memory compilation which is much faster than reading and writing to disk, but of course you are restricted to using Java 6 which not everyone has upgraded to yet.

Unfortunately there is a much more serious problem. Sun, in their infinite wisdom, decided that compiling code was something only developers would want to do and so only bundle a concrete implementation with the JDK and not the JRE which means that applets and Web Start applications don't have access to a compiler.

Fortunately Eclipse provide the JDT Core Batch Compiler which includes a Java 6 complient compiler. This works great in standalone apps, allowing me to run Saxon from a JRE and still have all the functionallity I need.

However, I was trying to make Saxon available as a Web Start application and ran into a bit of a problem. There is something very strange with the class loading in the Eclipse compiler. It works fine in a standalone application but under Web Start the compiler can only see core Java classes which means I can't reference my own API from the classes I'm trying to compile and everything fails and I end up back at square one.

Don't despair though as their is a solution. Throw away the Java 6 API and use the Eclipse compiler directly. This has the advantage of working under Web Start and under previous Java versions (at least version 5 which is more commonly installed than 6). I can't take all the credit for this solution but I thought it worth blogging about as I'm yet to see anything similar any where else on the web, although I did find lots of posts asking how to do this.

The following code is cobbled together from two main sources; Apache Tomcat and GATE. I actually used the GATE code, which is based on Tomcat code, as my starting point and then simplified it and added some generics support.

You can download the full compiler (a single Java source file and the JDT Batch Compiler) from here. And here is a simple example of how to use it (this is actually the main method of the class for easy testing).
EclipseCompiler compiler = new EclipseCompiler();

String HELLO_WORLD_CLASS_NAME = "HelloWorldTest";
String HELLO_WORLD_SOURCE = "import org.eclipse.jdt.internal.compiler.Compiler;\n\n" +
"public class "+HELLO_WORLD_CLASS_NAME+"{\n" +
" public static void main(String[] args) {\n" +
" System.out.println(\"Hello world\");\n" +
" }\n" +

Class compiledClass = compiler.compile(HELLO_WORLD_CLASS_NAME, HELLO_WORLD_SOURCE);

if (compiledClass==null)
System.out.println("Unable to compile file");

Method m = compiledClass.getMethod("main",String[].class);
m.invoke(null, new Object[]{null});
The interesting thing to note is that the small test case imports a class not from the core Java APIs and this works in standalone applications and when run under Web Start.
19 October 2008 at 10:27 , Morten Nobel-Jørgensen said...

Hi Mark

Thank you so much for EclipseCompiler-class. Finally I'm able to compile Java code in Web Start using Eclipse Compiler.
Btw. I think you should provide the full source code of EclipseCompiler class in your blog (as collapsed code using the syntax-highlighter).

- Morten

20 October 2008 at 20:19 , Mark said...

Thanks Morten, glad to hear you think it will be helpful.

As for putting the whole code on the blog, I did originally do this. Unfortunately I got lots of script timeout errors from the syntax-highlighter. I'm guessing it was just too long to be sensibly formatted.

18 February 2009 at 07:47 , pezzacolori said...

Hello Mark,
your post is really excellent ... it is a while that I'm finding a solution to my problems ... and your compiler nearly solve it ... but not completely.
I need to compile on-the-fly (in memory) new classes which holds database queries (to object databases using native queries) or functions in a static method. And I need to use it in a web application. I tried :
-Janino compiler> lighweighted, works well in swing and in web application, but do not support geenrics
-JDK compiler> works perfectely in swing application, but in web (tomcat) shows the same problem you told about WebStart applications ... the project packages are not recongnised during compile time.
-your Ecplise compiler> works well in both applications! ... but I can not compile twice the same class, since I get following error:
"java.lang.LinkageError: loader (instance of .../EclipseCompiler$EclipseClassLoader): attempted duplicate class definition for name: ...CookedQuery"

This works with Janino and Jdk compiler, where I can load several times classes with the same name.

Is your Ecplise compiler writing the classes to files?
Do you think there is a chance to get rid of this behaviour? Or maybe to apply your solution to the standard Jdk compiler?

Thank you a lot for your patience.
Kind regards,


22 February 2009 at 08:52 , Mark said...

Hi Boris,

The reason you can't reload the classes is the classloader. Basically a normal classloader will only ever load one instance of a class. I assume the new JDK compiler is doing something clever to flush the old version and replace it.

I'll look into trying to figure out how I could replicate the behaviour in this compiler.

In the meantime the solution I use is to ensure that when you try to recompile a class you use a unique name for every version but I don't know if that would be feasible in your code.

23 February 2009 at 23:35 , pezzacolori said...

Hi Marc,
your words gave me an idea ... and indeed I could solve it ... at least I hope it:
I simply set the _classloader instance of the EclipseCompiler not to be static:
private /*static*/ EclipseClassLoader _classloader = new EclipseClassLoader();

so that for each EclipseCompiler instance I have a different class loader ...
Pleas let me know if you think that this solution may have some problems or drawbacks.

Thank you again,

25 March 2009 at 09:09 , Laszlo Sas said...


check this article:


25 March 2009 at 09:16 , Mark said...

Thanks Laszlo, that looks like exactly what Boris was needing. I'll try and find some time soon to try it out and possibly merge it with the code I have so we can have dynamic class compilation and reloading in Java Web Start.

Post a Comment