Winter Has Arrived!

Well winter definitely arrived this weekend. On Saturday morning it snowed in Penistone, and it was so cold this morning that I've had to wear a jumper to work for the first time this year!

Hopefully there won't be quite as much winter as there was last year.

Forcing Rats To Eat Selenium!

As I've mentioned before we are currently in the run up to a big software release at work. Not only are we making a major release of GATE but we are also updating the website and making minor releases of other bits of software developed by the group.

As part of this I've been fixing bugs and looking at security issues in our wiki/CMS system -- GATEWiki. Testing a web application automatically and reliably can be really tricky, but we are using Selenium to test GATEWiki. Without going into too many details a Selenium test case is a script which is used to control a web browser (in our case Firefox) in the same way a normal user would interact with the application. We have Hudson setup to run a whole bunch of Selenium tests every time someone checks in changes (as well as running normal unit and integration tests), and anyone can check on the latest build results.

Whilst we have have been careful about security during the development of GATEWiki we want to make sure that there are no gaping security holes. One interesting approach to looking for security issues is to use ratproxy. As it's name suggest ratproxy is a web proxy that monitors requests for security issues. It is designed to watch normal user behaviour and so I decided that using it to monitor the Selenium tests being run would be a clever way of reviewing potential security problems in a repeated and reproducible fashion. Configuring this was actually quite straightforward (if you are interested I created a simple Firefox profile template directory with the necessary prefs.js file).

One obvious security issue was that we were not enforcing the use of HTTPS during login and hence passwords were being sent in the clear. GATEWiki is built using Grails which uses an embedded Jetty web server and it was easy to configure this to send all requests that would involve sensitive data via HTTPS. Unfortunately this caused Hudson to be unable to run the Selenium tests via ratproxy.

The problem is that when secure requests are sent via ratproxy, ratproxy throws away the original security certificate and re-signs the requests using it's own certificate. Due to the way Firefox now handles untrusted certificates (lots and lots of big scary warnings) there is no way I could find to make Firefox remember that it should trust pages signed by ratproxy and hence it would just sit there waiting for the user to accept the certificate. This is fine for local testing, but not much help for unattended testing via Hudson. Fortunately the fix is actually quite easy, although it took me an age to figure out.

If you look in the ratproxy directory you will find a file called keyfile.pem which contains the security certificates used to sign any secure page. All you need to do is replace this file with one that a) ratproxy can handle and b) Firefox can be made to remember should be trusted.

If you are on a Linux box or have cygwin installed under Windows then you probably have a copy of the OpenSSL package available and this can be used to produce a new set of certificates which will do the job. The keyfile we are trying to produce actually contains both a private key and a public certificate so we need to create both of these and then combine them into a single file.

Let's start with the private key as we need this to create the certificate. Simply issue the command:

openssl genrsa -out keyfile.key

Dead easy! The thing to note is that this key isn't password protected, which may not be ideal but is required otherwise ratproxy won't be able to use it.

So we have the private key let's now create the certificate. This is a bit more complex but start by issuing the command:

openssl req -new -x509 -key keyfile.key -out keyfile.crt

Now this will ask you for a whole bunch of information. Fortunately we don't have to provide many answers. Simply leave all the fields blank (by answering .) except the 'Common Name' field. This must be set to the hostname of the ratproxy server. Most of the time this will simply be 'localhost' but if you are running the proxy on a remote machine then you will need to set this value appropriately.

The final step is to combine the key and certificate into a single file that you can use to replace the default ratproxy keyfile.pem file with. Just issue the command:

cat keyfile.key keyfile.crt > keyfile.pem

And that is it. You now have a file that ratproxy can use and that you can tell Firefox to trust permanently.

I know it probably isn't an overly interesting blog post (I certainly doubt it lived up to it's title) but hopefully it will save someone else an afternoon of trying to figure out what's wrong with the ratproxy certificate and how to replace it.

Fussy Eaters

I'm sure that most of us know at least one fussy eater. You know the kind of people; I don't eat anything that's green; I don't like oranges; salad, not a chance!

When I was growing up my brother was the fussy eater (sorry David but you know it is true), whilst I'd eat anything you put in front of me. There are of course things that I'll eat because someone else has cooked, but that I don't particularly like. Specifically I don't like sprouts. No matter if they are boiled or steamed they just turn out inedible. But of course it isn't the food but how it is cooked that is usually the problem.

Take my brother for example, the only type of potato he would eat as a child was chips. My Mum refused to always cook him chips but we found that if we steamed or boiled potatoes then sliced them up and stir fried them for a few minutes he would eat them as they kinda looked and tasted a bit like chips. Well it turns out if you stir-fry sprouts, following this recipe, they also become an edible food-stuff -- although I'm guessing the shallots and Parma ham help!

Comment Spam

I've been writing this blog for over two years now and have never suffered from comment spam until recently. I'm not sure what has changed but I've started getting really obvious (or at least I think it is obvious) comment spam.

A person either without a public profile or who has just joined blogger will comment on two or three posts within a matter of minutes. The comments are usually only slightly related to the post and sometimes not at all. But what they all have in common is a link at the end of the post to some random web site; usually the site seems, at first glance, to be a blog or encyclopedia but when you look for more than a few seconds it is clearly trying to sell things without making it too obvious.

I was just ignoring them and hoping I wouldn't have to hurt anyone's feelings by deleting the comments but then the other day someone posted the following comment in reply to my post on Pomegranate, Cranberry and Pepper Jelly:

Credit unions are recognized as a force for positive economic and social change and have provided significant social and economic value in both developed and emerging nations. International Credit Union Day is sponsored by World Council of Credit Unions (WOCCU), the international trade association and development organization for credit unions, and Credit Union National Association (CUNA), the premier trade association for America’s credit unions.

Now that is definitely spam -- there is no way in which it is related to the post and it is definitely unwanted. So that was the final straw, anything that looks like comment spam will now be deleted as soon as I see it. I'm not going to turn comment moderation on yet as hopefully I'll be able to spot and delete them quite quickly.

If you are reading this and have one of your comments deleted then either a) it was spam please don't do it again or b) I deleted it be mistake whilst annoyed at some other spam, sorry!

The Pain of Flat Pack Furniture

Flat pack furniture is great; you don't have to worry if it will fit through the door, you can easily move pieces around to the right room to assemble, and it's usually cheep but functional.

The pain, both figuratively and physically, comes when it takes six, yes six, hours to assemble a single item of furniture! No I'm not incompetent! It was a big sideboard and one of the pieces had been drilled the wrong way around necessitating a 20 minute detour from the instructions while I disassembled instead of assembled the base unit.

I put it together Sunday afternoon and my hands, legs and back still haven't recovered from the experience. The palms of my hands are actually bruised from gripping the screwdriver.

Maybe next time it would be easier just to buy ready assembled furniture after all!

Pomegranate, Cranberry and Pepper Jelly

Some of you may remember a previous post in which I mentioned the difficulty in finding an ingredient. I had found that sauce made from red wine and pomegranate, pepper and cranberry relish went really well with venison steaks, but that I couldn't find any more of the relish. Well my jar of relish finally ran out and I still haven't been able to buy anymore. So last weekend I decided that I'd just have to make an alternative.

I hunted around the web but couldn't find an appropriate recipe so I took suggestions from quite a few places to come up with a recipe for Pomegranate, Cranberry and Pepper Jelly.

Last night we tried the jelly in the sauce (served with beef rather than venison) and it was superb. It wasn't quite as hot as I would have liked (hence putting pepper last in the name), but I can tweak that a little when I make a future batch.

As an added bonus I had some cranberries left over so I decided to try and make my own Cranberry and Orange Muffins. Eaten warm from the oven they were superb. Even cold I think they are better than the one I bought in Starbucks!

The Trotternish Pumpkin

Question: What does the Trotternish peninsula of Skye and a pumpkin have in common?
Answer: The images in the two blog posts were assembled using PhotoGrid!

When I was writing the blog post about our trip around the Trotternish peninsula I looked high and low for software to easily create a grid of photos. I found a few options but they were either a) a feature in a huge piece of software or b) not able to create the kind of image I wanted. Of course this could mean only one thing... I'd have to write the software myself.

The image in the Trotternish blog post was from the first, very basic version of what I'm now calling PhotoGrid -- it arranged the photos in a grid with the only option being the number of rows. The software has advanced quite a bit since then as you can tell if you look at the image in the pumpkin post. Here is a screenshot of the latest version:

There are still lots of things that I would like the program to do that it doesn't but hopefully I'll find the time to add new features soon. For now at least the main features are:

• Auto-arrange will attempt to separate images of a similar colour to produce a more appealing grid -- for example, if your photos are either mostly white or mostly black you will end up with a chequer board pattern.
• The border colour is determined by working out the average colour of the photo as this is unlikely to clash with the photo.

Eventually I'll clean up the code and release it but for now you can and use the software to create your own grids.

If you have any comments/suggestions about PhotoGrid or ideas for future versions then please leave a comment and I'll see what I can do.

---

v1.0.0 - 19/08/2010:Lots of bug fixes and performance improvements. Some new artwork and better error handling. There were some changes to the way in which the application can be started from this page. If you have used the application before and now it won't start, sorry, but see this post for details on how to fix things.

v0.4.0 - 06/01/2010:Almost no code changes but this release coincides with the opening up of the source code.

v0.3.1 - 30/01/2010:Quite a few performance improvements and real feedback on progress of loading images or saved projects. You can also now choose auto to reset the border colour to the default.

v0.3.0 - 24/01/2010:You can now customize each tile (or a set of tiles) by right clicking on them. This allows you to set the border colour and specify a title and URL which can be used to produce a HTML image map. Also when saving the grid you now have some control over the size of the generated image.

v0.2.2 - 20/01/2010:Fixed a small bug in v0.2.1 and some serious refactoring that will make the new features easier t0 implement.

v0.2.1 - 18/01/2010:More code simplification, added an extra theme colour (the average colour across all photos), and added some code in preparation for new features I'm intending to add soon.

v0.2.0 - 12/11/2009: Simplified a lot of the code but more importantly removed the assumption that images were always 4:3. Each tile is now assumed to have the same aspect ratio as the first tile in the grid.

v0.1.1 - 10/11/2009: The only change in this release is to use all known JPEG file extensions (ignoring case) when filtering the file list whilst trying to add photos.

v0.1.0 - 09/11/2009: First public release of PhotoGrid.

It's Christmas.... At Starbucks

I hate the way that holiday treats are no longer confined to their holiday. For example, a Cadbury Creme Egg isn't a special Easter treat if I can buy them the first week of January. I have, however, been waiting since the first week of January for Starbucks to think it's Christmas!

Yesterday I walked into Starbucks and noticed that the usual paper cups had been replaced by red cups with snowflakes ready for Christmas. Why is this important? Christmas means they sell Cranberry and Orange Muffins! If you haven't tried one before then you really should stop whatever you are doing and find your nearest Starbucks as soon as humanly possible -- they really are that good.

(Werid Note: Starbucks is in the dictionary Blogger uses but Cadbury isn't. I wonder how much Starbucks coffee the Blogger staff drink?)

The Line Endings Are Conspiring Against Me!

It's not paranoia if they're really out to get you!

At work we are gearing up for an important release of GATE. We released a beta last week so that we could get as much feedback as possible on the current build before making the final release. Unfortunately I managed to introduce a show-stopping bug into the build which rendered the default GATE application (ANNIE) unusable under Windows.

I'm pretty careful about testing code before I commit changes so I was mortified when it became obvious that the beta was badly broken. What follows is a verbatim copy of an e-mail I sent to the GATE group explaining how the bug was introduced and why it hadn't been spotted by any of our tests. The specific situation might not happen to many software developers but it's a good example of why we shouldn't rely only on automated tests for application stability.

---

I was very annoyed at myself for allowing the bug to slip into the beta build but at a complete loss as to exactly how it happened. I've been thinking about it some more and I now know how it happened and I think the reason is important enough to share with everyone.

This isn't an excuse for what happened but it should be a warning to everyone else that sometimes even when we are careful with tests etc we can get caught out. This will be quite a long e-mail as I want to fully explain the situation for those not familiar with the internals of the JAPE parser/compiler. If you want the take home message there is a simple summary at the bottom.

The code that fails was trying to split a block of Java code generated by parsing a JAPE file into separate lines and then return the line at which use of the deprecated annotations parameter had been spotted. The method I checked in looked like this:

public String getSource(int line) {
 String[] lines = getSource().split(nl);
 return lines[line-japeLine];
}

where nl is a class field that is initialized to System.getProperty("line.separator"). So this method should split a block of text into separate lines based on the platform specific line separator and then return a specific line from the resultant array. This should be safe as the code block is from the generated Java source, which uses the same nl field for separating lines.

I developed the code at home under Windows and it worked -- ANNIE ran with no sign of the exceptions reported against the beta. I checked the code in and a short while later Hudson checked it out, built it and tested it (including running ANNIE) under Linux (see here for the latest build/test results). Everything built and ran successfully.

At this point with it having been tested under Windows and Linux I was fairly happy that the code was stable (I was especially worried after Hamish had agreed to the change but had cautioned about adding new code so near to a release as statistically new code means new bugs - man was he right!).

Ian then built the beta release, publicized it on the mailing list and as we know ANNIE broke spectacularly for anyone trying it under Windows. The only question is why?

I'm willing to bet a beer/coffee to the first person with the time to checkout and build the beta from subversion (SVN) under Windows that it works. So what is different about the builds Ian pushed to the website?

The answer is that the line endings and subversion have conspired against me :(

When a JAPE source file is parsed the Java blocks that make up the RHS are added to the Java source code as is. And we use the platform specific line separator to build the rest of the source. When we check JAPE files into SVN we make sure that we set the svn:eol-style to native. So if you check out GATE from subversion all the ANNIE JAPE files have the native line endings and so the full Java source file that we eventually compile has the same line ending throughout and everything works. This is why ANNIE ran under both Windows and Linux when checked out of SVN.

When Ian built the betas I'm guessing that he did so under Linux (or on his Mac but for the purpose of this discussion that doesn't matter). So again the tests would run as the JAPE files would have Linux line endings and we would use the Linux line separator.

A user then downloads the beta from the website and tries running ANNIE under Windows -- it fails. The problem here is that the JAPE files in the beta builds have Linux line endings and we then use Windows line endings to assemble the code. I then use the Windows line separator again to split the code to get at single lines of source. Linux uses the single \n to represent the end of line while Windows uses the two character \r\n. So when I try and split code containing Linux line endings using the two character Windows line ending nothing happens and the array offset exception is thrown.

If we had built the beta under Windows then we may never have spotted the problem, as splitting on either platform using either line ending would have worked. The problem would have only arisen if someone created a JAPE file under Linux and then tried to use it under Windows without SVN in the middle, which conceivably might not have been until after the final release.

My fix was to simply change the offending method to

public String getSource(int line) {
 String[] lines = getSource().split("\n");
 return lines[line-japeLine];
}

which works on all platforms we support.

Looking back it's clear that there is no way we could have easily caught this using the automated tests. Fortunately we have caught the problem now rather than after the final release, yet I don't have any idea how we could stop similar problems occurring again in the future.

In Summary: The bug only appeared because the svn:eol-style was set to native, the beta release was built under Linux, and together this means we end up with a mixture of line endings when running under Windows. Had we built the installers under Windows the issue of running ANNIE would never have arisen but it would have bitten anyone editing JAPE under Linux then running under Windows (without SVN in the middle), which may not have happened until after the final release.

If anyone has any thoughts/suggestions on ways we could improve the testing to try and pick up such weird cases in the future please let me know and I'll try adding them to Hudson.

Sorry for both the bug and the long post but I thought it worth taking the time to explain how the problem arose so we can all (me especially) try and avoiding it happening again in the future,

Measuring Sheffield

I've lived and worked in Sheffield for 12 years now and yet I still sometimes come across things that I've never noticed before. For example, last Thursday I walked down the side of the Town Hall and spotted this set of standard measures.

In fact there was a lot more than just this one plaque as the floor has small plaques set into it to show longer measures.

Apparently the standard measures were "presented to the city by The Right Honourable The Lord Mayor (The Earl Fitzwilliam, D.S.O.) and by him declared available for public use on the occasion of the visit of the British Association, September, 1910".