Better Search Results

Having added search to the blog last week I had a further play around with it and noticed that certain searches were not returning the results I expected. So I dug a little deeper into the documentation for Google Custom Search to try and figure out why.

Whilst the documentation suggests that Google try and build an index of 10,000 pages for each custom search engine they start by using just the pages already in the main Google index. The problem of course was that not all the pages from this blog were in the main index and so they were not showing up in the results of searches using the custom search either.

Now as Rob pointed out in a comment on the previous post the way to tell Google (and most other search engines) about the pages on your site is via a sitemap. A sitemap is an XML file that basically lists all the URLs within your website, how important each URL is, when the content of each URL last changed and how often the content of each URL is likely to change. You can find full details of the file format and protocol at the official sitemaps homepage. Whilst the file format is very straightforward collecting together all the information is a little more tricky.

It turns out that Google will accept RSS feeds as well as normal sitemaps so I could just associate the RSS feed that Blogger generates with the custom search engine but that wouldn't allow me full control over the sitemap or to tell other search engines about this blog. Instead I've written a small tool, SitemapGenerator, which when given a folder structure containing a copy of a website will generate a sitemap. I won't go into the details here as I'm sure they would bore most of you, but in essence the tool uses regular expressions to allow you to assign different importance and update frequencies to different pages within the site. The tool can also automatically notify a number of different search engines, including Google, of any changes to your sitemap.

Now sitemaps make updating the pages used by a Google custom search engine really easy. I have an automatic process in place that checks this blog for changes every hour and if needed will regenerate the sitemap and notify Google of the changes. This meant that only 25 minutes after yesterday's post searching for the word daemon was returning the new post.

Of course sitemaps aren't just used for building better custom search engines, they are also used in the generation of the main Google index (and the indexes of the other major search engines). So if you have a website and are worried that it isn't being properly indexed, then build a sitemap and register it either via my tool or via Google's Webmaster Tools page (if you want to get stats about the effectiveness of the sitemap then register it via the web page before allowing the program to notify Google). For example, not only did yesterday's post appear in the results of my custom search engine within 25 minutes but the page is also appeared in the main Google index at about the same time.

So if you have read this far you might well be interested in having a play with the SitemapGenertator so download it have a look at the README file for instructions and then start making sure your website is properly indexed by the main search engines. And of course if you have any problems with the software or suggestions for improvements then leave me a comment.

Search This Blog

The eagle-eyed amongst you may have spotted that I've added a search box to the blog (top right if you haven't spotted it yet).

I've got frustrated a few times recently when trying to find old posts so that I could link to them in new articles and so had started to think about adding a search feature. A request from a reader (thanks GB) provided the extra kick of motivation to finally get me to sort out a proper solution.

Now I could easily have added search to the blog by enabling the blogger navbar but I don't like the navbar for two reasons: it looks terrible but more importantly the search feature is weird.

The search provided by the navbar is limited to just searching the content of each post (and I assume the title) but not the labels or comments which can make finding old posts more difficult than it should be. So I decided to try a different approach.

Google provide a rather cool custom search service. This allows you to build your own search engine by providing filters that select just a subset of Google's main web index. The simplest option for creating a custom search engine for a blog is to provide a single filter that selects everything. So for this blog I could have used www.dcs.shef.ac.uk/~mark/blog/* as the filter. Whilst this is easy the downside is that you get a lot of repetition in the search results. Remember that each post appears on it's own page as well as on the monthly archive page and the page for each of the labels it is tagged with. To get around this problem I'm actually using three filters to select just the post pages. I need three filters as I've posted articles in three different years (2007, 2008 and 2009). So the first filter is www.dcs.shef.ac.uk/~mark/blog/2007/* and I'm sure you can guess at the other two. Of course this solution isn't perfect either. Firstly when we move into 2010 I have to remember to add a new filter but secondly the whole page is now being indexed which again can lead to repetitive search results. For example, only a few posts contain the word sugar (4 I think) but it appears on every page as it is in the blog description. Fortunately Google is quite good at filtering these useless results out as you can see here.

There are quite a few options available for customizing the search engine so I may fine tune things later but for now at least I have a useful search tool for me and my readers. If you spot anything weird or have any suggestions then please leave me a comment.

If you have been annoyed at the way the standard navbar search works then I'd certainly have a play with the custom search service. Of course there are no limits on what pages you can include in the search. You could create a search engine which indexes multiple blogs or pages you frequently visit. One thing I have noticed though is that if you change any settings (or when you create the search engine) it can take five or ten minutes before the changes take affect so don't be surprised if it doesn't work straight away -- I couldn't understand why I got no results for any search to start with but after about five minutes it started to work just as I had expected.