Christopher Hunt on Software Development

Application developers should not write frameworks and toolkits (much)

2011-09-08T07:40:00.001+10:00

On occasion I'm a little disappointed when I see developers create frameworks and toolkits. I understand that it is enjoyable to do so, but we, as application developers, should be majorly focused on the business needs of developing applications.

Quite often we think that xyz widget doesn't do exactly what is "required" and therefore pursue a path of days (weeks) writing frameworks and toolkits.

Let's not do this anymore.

Instead, every time you, as a developer, get the urge to write a framework or toolkit or even just a utility class, think about extending an existing open source project to meet your needs; or even starting a new open source project. For one thing, your requirements will most likely be challenged (which is a good thing). You will no doubt end up with something better than you could have created yourself (yes, that's hard to swallow, I know!). Importantly though, your organisation will benefit; the less "quirks" particular to an organisation the faster it is for them to ramp up resources and "Get Things Done".

Your organisation will also be able to share stuff between the silos. Imagine that.

I often hear, "but my organisation doesn't contribute to open source" (despite heavily using it). In this instance I always ask if the person in question has actually tried selling the business benefit (above) and attempted to make it happen. More often than not, the developer hasn't tried. Organisational behaviour is driven by the views of their employees and contractors. Organisations change and are changing.

I do believe that at the end of the day, it is better to suffer with an xyz widget that does most of what you need it to do than to re-invent the wheel. If you're an application developer then focus on the application, not frameworks and toolkits; unless you're contributing to open source projects.

Dependency Injection in Java - @Resource, @Inject or @Autowired?

2011-08-11T13:28:00.001+10:00

The @Resource documentation states that the "name" attribute refers to "The JNDI name of the resource". Spring has overloaded this to mean that you can refer to bean identifiers… but as far as the Java spec goes, there's no contractual obligation for a resource name to refer to a bean id. @Resource referring to Spring bean ids is therefore quite Spring-specific.

@Autowired is Spring specific.

@Inject is not Spring specific and performs type-safe injection.

The other benefit you'll get from type-safe injection is the ability to safely re-factor within your favourite IDE, but I personally think that the contractual obligation mentioned above is more important.

Versioning RESTful services

2011-08-10T12:45:00.001+10:00

I've seen some comment out there regarding how to version RESTful services and so thought it about time to add in my own position on this.

I don't subscribe to crafting your own MIME type as these should refer to well-known entities. I also don't subscribe to passing a version number as an Accept param. Despite there being client/server compatibilities in this regard (Ruby), the developer is not forced to specify the version. I think it is useful to specify the version of a service to use.

I'm an advocate of using a /v1 path in the url of a RESTful service, but it should be noted that great care should be taken in exactly what that version number refers to. I've long subscribed to the major.minor[.maintenance[-build]] approach as per Maven and reasonably discussed at Wikipedia. In the case of URL versioning, only the major identification is used i.e. "1.0.2" becomes just "v1".

Major versions only change when there is an API-breaking change that has been introduced. This should be very rare. API-breaking generally means quite a big shift in the approach an API is taking which is why I found it interesting that Google declare their Google Maps API will always be backwardly compatible. To me that's effectively what held Microsoft back with Windows until Vista came along and started to drop backwards compatibility (Windows 7 of course is the evolution of that good decision).

So I say use /v[major] as a convention, but be diligent in the definition of what is "major".

LMAX and the benefits of keeping things in memory

2011-07-22T22:36:00.001+10:00

I've now read Martin Fowler's review of the LMAX architecture and I feel good. This is very similar to what I've been doing for a long time with messaging (Spring Integration, Camel, AMQ, RMQ etc.).

LMAX is an event driven architecture. The Input Disruptor can be your favourite messaging broker. The Business Logic Processors are your message queue consumers. AMQP can certainly attain the goals described for the Output Disruptor (targeted consumers etc.).

The only thing "disruptive" about this article is that it introduces new terminology. I accept that there's probably a lot of optimisation going on, but from an architectural standpoint I don't see anything revolutionary.

Perhaps the most interesting points of the LMAX architecture are that everything is done in memory and that the processing of data is kept close to the data. These are things that we will all no doubt agree on being the main influence on performance.

RESTful JSON fixtures for testing using CouchDB

2011-06-26T17:31:00.001+10:00

I'm working in a distributed team at the moment where I don't have access to infrastructure available within the company's firewall. The application I'm working on has a nice RESTful API with JSON payloads for its logic layer. As I'm mostly developing the presentation layer I needed a quick method of generating something that would simulate the logic layer.

One thing that I felt was important is that I shouldn't have to mutate any settings within my presentation application in order to work either against the simulated logic layer, or the real one. It is all too easy for such settings to creep into production and break things.

I had a few thoughts on how to achieve my objectives considering Spring MVC,JAX-RS re-implementation using our existing interfaces (we're using JAX-RS in the real logic layer), writing a Jetty based servlet handler and node.js. Then I remembered...

CouchDB is a document oriented database that uses JSON as its Data Description Language (DDL). In addition CouchDB provides a RESTful JSON based API to access the database. This got me thinking that I could use sample responses from our real logic layer and enter them directly into the CouchDB database. It turns out you can using the batch interface!

I then thought that I needed to create a CouchDB service that could understand the parameters of the web service request I needed to make. These parameters are properties of my JSON objects that are otherwise unavailable to the URL format that CouchDB provides for querying. The answer here is to provide a View. Here's my view:

function(doc) { emit(doc.surname, doc) }

The surname is a property of my JSON object that I'll want to query.

I call the view the same name as my logic layer service. In my case the view's name is "getCustomerBySurname". If I query my service using CouchDB's URL convention I would issue something like:

http://localhost:5984/customerdb/_design
/customerService-1.0/_view
/getCustomerBySurname?key=%22Hunt%22

This yields a result like:

{"total_rows":1,"offset":0,"rows":[
  {"id":"285b217949d54c29c4b27adf6d000da2","key":"Hunt","value":
    {"id":1,"surname":"Hunt","firstname":"Christopher"...

Note that I'm using a convention for versioning a resource so that I can evolve its API nicely in the future - this is quite important for RESTful requests as versioning isn't something supported by anything within HTTP

I now have a web service that behaves the way the real service behaves. However, it doesn't look the same to the consumer for two reasons:

the result contains meta information about the query that the consumer does not expect ("total_rows" etc.); and
the URL we provide is not of the same form as the real logic layer service.

Transforming the JSON result

To transform the payload returned by CouchDB into something our consumer expects we create a List in CouchDB-speak. My list function looks like this:

function (head, req) {
  provides('json', function() {
    var results = [];
    while (row = getRow()) { results.push(row.value); }
    send(JSON.stringify(results));
  });
}

I've declared my list by the name of "rows". The function is focused on pulling out just the row.value property of each row and returning that back to the consumer. To access the view that I created and have it render through the list you provide a URL like this:

http://localhost:5984/customerDB/_design/
customerService-1.0/_list/rows/
getCustomerBySurname?key=%22Hunt%22

This now yields something like:

[{"id":1,"surname":"Hunt","firstname":"Christopher"...

...which is what my consumer expects.

Transforming the url

The url is transformed via Apache httpd using a RewriteRule. In order to avoid Same Origin Policy we've had to proxy requests for our presentation component and its web services anyhow. Here is the re-write rule I ended up with:

RewriteEngine On
RewriteOptions Inherit
RewriteRule ^/CustomerDB/services-1.0/rest/Customers?surname=(.*) http://localhost:5984/customerdb/_design/customerService-1.0/_view/getCustomerBySurname?key="$1" [QSA,P]

... and that's it. I now have a means of using CouchDB to provide fixtures for the purposes of development.

One last important thought...

The really interesting thing about using CouchDB for the use-case I set out to solve is that I'm really starting to wonder whether the real logic layer should be written using CouchDB... it is so simple and powerful!

DAO coding paradigms can require some skill

2011-04-08T21:39:00.001+10:00

"DAO coding paradigms can require some skill" is a sentence that presently appears at Wikipedia in the description of Data Access Objects. I find DAOs very useful and honing the skill is worth it. The article goes on to state a benefit being, "Improved efficiency and performance of the data layer since it is standard reusable software".

The majority of DAO implementations that I see tend to provide similar functionality to the Active Record pattern i.e. CRUD. Unfortunately these DAOs do not achieve the benefit of improved efficiency and performance. Let's say I have a customer, with orders that relate to products. My use-case is that I want the customers along with the orders and their products. With a straight CRUD approach I tend to get the customers (c) and then I get the orders (o) and then I get the products (p). That's potentially c * o * p or more requests on the database over the network.

The method signatures of my DAOs are shaped by the service layer that drives them. Here are a few methods of one of my real-world DAOs to serve as an example:

public void createSMSSubscription(
    String msn, String flightNo,
    String actualText, String receivedParsedText, 
    Date scheduled, Date eventTime, 
    AbstractSMSEvent subscriptionEvent) {

public Collection<Subscription> getActiveSubscriptions(
    AbstractRenderedFlight renderedFlight) {

public Collection<ReceivedSMSFlightSubscription>
    getCurrentReceivedSubscriptions(
        String msn, Date receivedTime) {

It may be that the service layer simply asks for a Customer to be created, but more often than not, the methods tend to be tailored to what the service layer wants to do e.g. "create an order for a customer that exists and only for products that exist", or "create an order for a customer that has not exceeded their credit limit" and so forth. I like to have my DAOs do as much as possible at the database with the objective of minimising the number of calls made to it. In short, my DAOs tend to comprise of fairly coarsely grained functions instead of finely grained CRUD style operations. I'll admit that it is a bit of an art in getting the granularity correct and that re-factoring commonly occurs, but the overriding goal is to reduce the IO between my logic layer and the data layer. Performance and efficiency then typically follow.

My DAOs are developed as the service layer evolves i.e. they are not developed before the service layer. As stated above I let the service layer determine what functionality the DAO should provide. If I've developed a service with the DAO development alongside, and then develop another service that wants more out of the DAO then I tend to done one of two things:

parameterise the existing methods of a DAO such that their behaviour can be varied; or
create new methods on the DAO.

One more thing while I'm thinking about DAOs... they're a great place to encapsulate persistence functionality of course. It used to be that we may have wanted to hide away the differences of dealing with multiple types of relational database. However we now find ourselves thinking about providing non-relational stores (NoSQL) as well. DAOs give you the freedom to choose the best persistence implementation to suit your application's requirements.

Long live the DAO!

Zoneinfo TZ v1.0.0 is now available!

2011-04-03T02:22:00.000+10:00

My Zoneinfo TZ library is now available up at the Codehaus. V1.0.0 should make it to the central Maven repo shortly. The project has mailing lists, continuous integration, issue tracking and all of those things you'd come to expect.

Interestingly, the site that hosts the zoneinfo files themselves has disappeared tonight. Fingers crossed that it'll come back soon!

I hope that this project proves useful to someone other than me! Enjoy.

GWT and GXT

2011-03-02T09:18:00.001+11:00

Recently I've been doing a lot of development in GWT. I have to say that, despite being a big fan of pure JavaScript RIA, I'm liking GWT a lot. It has been a couple of years since I touched GWT and it has come a long way. I particularly like the Activities and Places API, although there's a bit of a learning curve there.

One thing I don't particularly like though is Sencha's EXT-GWT, or GXT as it is known. For those that don't know, GXT provides an abundance of widgets including a fully featured Grid. This is of course highly enticing, but what I've found is that GXT appears to be an all-or-nothing proposition. GXT is more akin to a framework than a toolkit because it effects how you lay out your code. What isn't clear to me is why it has its own layout architecture and its own event architecture. That's just confusing!

I'm starting to think that GXT plugged a few gaps in GWT 1.x and is becoming increasing less relevant. GWT 2.x deprecates a lot of GXT; even GWT 1.6 introduced a new event model.

Don't even get me started on GXT's Grid separation of concerns; its one of those widgets that likes to influence how the data transfer objects look. Plain wrong. The whole loader concept is terribly confusing as well.

I feel that if GWT's TableCell was as fully featured as GXT's Grid then no one would probably use GXT.

Perhaps I'm being a little harsh though. GXT widgets present nicely. It is just that they don't just "slot in"; you've got to buy into all of GXT to use a little of it.

I have a new job

2011-01-13T10:13:00.001+11:00

I now have a job with VMware starting January 24th and I'm excited! My role is as a vFabric consulting software engineer within the professional services division .

Over the past 12 years I've been an independent contracting developer/architecture/designer and launched quite a few of my own products. I suppose one of the initial drivers to go contracting for me back then was that it was lucrative to do so. I now feel that the gap in remuneration between going permanent and contracting has diminished.

Another incredibly important factor was my family. Contracting can be great but the cash flow can be stressful. With 4 kids I needed more stability.

Finally, I wanted to work with more like-minded people. During my contracting years I've met some truly inspirational people along the way; but I want to meet more! This can be difficult as a contractor as you tend not to stay around long enough to work on the strategic projects and with an organisation's best people.

So, back in May last year I decided to look around for some permanent work. I choose some companies with SpringSource being one of them. I thought, if I go for a permanent job then who would I really want to work for? It is important to "tow the corporate line" of course and so I needed to ensure that this would be a no-brainer. I'm not very good at lying!

I've been using the Spring framework for quite some time and became very interested in the work of some of the SpringSource people; namely Rod Johnson and Ben Alex. Towing the line for SpringSource would not be a difficult proposition... I contacted SpringSource and ended up meeting with Guy Baldwin and the ball then started to roll. I instantly liked Guy and his technical knowledge along with his amiable personality makes him a great face of VMware/SpringSource.

'better had just get one thing straight: VMware acquired SpringSource in 2009 and so SpringSource is now a division of VMware. I'll refer to SpringSource as VMware for the rest of this blog ('hope that is politically correct!).

vFabric is the name given to VMware's Cloud Application Platform, meaning the whole stack of technologies that enable applications to live in a cloud environment. Technically speaking the cloud may not be the actual cloud as in the internet, but live behind the corporate firewall; it doesn't really matter though. What the platform is really about is enabling applications to be scaled in order to meet unanticipated demand.

My specific role is a first here in Asia/Pac. VMware have a vFabric consulting presence in the US and I'm their first consultant for them here in our region. It is exciting.

I'll continue to post my own views on software development here but needless to say, they'll be my views and not necessarily VMware's!

The Low Cloud

2011-01-03T21:19:00.001+11:00

I was chatting with a CIO mate today about the future of cloud computing and how his company are looking to build their own Cloud Enabled Application Platform (CEAP). I feel that many companies are sensitive to putting their data in the cloud itself and my friend definitely echoed that sentiment. There are some things his enterprise could put in to the cloud for economic reasons, but most things will reside behind their firewall.

Interestingly my friend's main driver for a CEAP was to achieve horizontal scalability. He has some performance issues that lend themselves well to being addressed in a highly parallel manner.

During this conversation I used the term "low cloud" to describe CEAP hosted behind the enterprise firewall. So, you heard the term, "low cloud" here first. :-)

Timezones, timezones and a new time zone library

2010-12-29T22:13:00.001+11:00

I've had a bit of a thing about time zones and daylights savings rules for quite some time i.e. they interest me. This was first apparent in an application I wrote for the Palm OS originally named Time Traveler (which became Titan Class). Its time zone database was inspired by zoneinfo, but was no where near as sophisticated. The Palm OS was without time zone support in its first few incarnations. Eventually time zone support was incorporated but IMHO it was weak. I digress a little... the point is I've really liked the zoneinfo database that underpins most Unix based platforms for quite some time.

Java supports zoneinfo in its implementation by use of the ICU library. There is also a JSR that's about 3 years old which aims to provide greater support for zoneinfo.

I recently had a need to ensure that my application had the latest time zone rules available. From a Java perspective, Sun or Apple (or whoever) provide a patch when time zone rules change. You then typically restart your application. I didn't want that. My goals were:

to have the latest rules on a monthly basis; and
to be able to dynamically update the time zones without having to restart my application.

JSR-310 could probably help me out here but I had another nagging concern; in fact a couple:

the JSR is 3 years old and doesn't appear to have progressed; and
I like the zoneinfo structure and wanted to use something that honoured its structure closely.

I might be a little unfair toward JSR-310 and if it becomes approved then it'll be difficult to avoid. I'm also strongly aware of the "Not Invented Here" syndrome... not my style though. Then, there's the JCP-is-dead thingy...

So, what I've done is created a new Java time zone library that takes zoneinfo files and produces a JDK compatible facade. The library uses ANTLR to parse the zoneinfo files thus actually providing a parser that can be used for many languages. I'll shortly be open-sourcing this library and probably at The Codehaus depending on how well it is received there. Meanwhile here is an overview of its structure:

The library is effectively done and has reasonable test coverage. I hope that you'll join me and help improve it. Meanwhile any thoughts and ideas are most welcome.

Memory grids

2010-12-07T21:52:00.001+11:00

There's a great podcast on Software Engineering Radio with Nati Shalom. The podcast discusses memory grids and I certainly found it insightful. In essence memory grids are being looked upon as the next disk.

The clincher for me was the revelation that the durability of data is not related to it being persisted to disk; it is related to the number of geographically disbursed copies of that data at any one time. Taken to the extreme this could mean that you don't need disk at all, but practically the data gets persisted to disk in an asynchronous manner. This is called "write behind" and can be performed at n nodes, if not all of them.

I think memory grids are very interesting. My prediction is that memcached, a popular open source memory cache that can be distributed over many nodes, will become a memory grid offering write-behind persistence. Same goes for Ehcache/Terracotta i.e. these memory caches will evolve beyond being just that. There are of course commercial memory caches out there including vmware's Gemfire and Oracle's Coherence.

One reason in my mind as to why memory grids are topical is commodity hardware being able to address one heck of a lot of memory resident data. Since the introduction of 64 bit computing for the masses, we now have a situation where a cheap processor can generally access about 256TB data - more than enough for most databases! Of course, with 32 bit processors about 4GB could be addressed which is less than many databases.

I think something that can be overlooked with memory grids is persistence. As mentioned, write-behind appears typical, but what isn't focused on is what performs the write-behind. There's no reason why that write-behind can't be done with a conventional RDBMS and I understand that many memory grids support such a thing. Thus with memory grids, it appears that you can have the best of both worlds.

Bring on the memory grid (preferably open sourced!).

Mac OS X Virtualisation

2010-11-06T21:09:00.001+11:00

This evening I read that Apple are retiring the Xserve. I guess that's a shame as I always wanted to see more of OS X in the server space. I think that this will relegate OS X to the SOHO market; unless... Apple permit virtualised instances of OS X.

Permitting virtualisation would be a very smart move by Apple. Let's face it, unless you can run on OS as a virtualised instance then you're never going to be a player in the enterprise and in the elastic cloud.

Time will tell if Apple see the merits of virtualisation.

JavaScript Dependency Management and JSLint Tools

2010-11-04T12:20:00.001+11:00

I've been focusing on JavaScript Rich Internet Application development for some time now and have felt that while JavaScript is a capable language, it has sorely lacked tooling. This situation is changing of course and we're seeing more JavaScript tools come from the likes of Mozilla, Apple and Google quite frequently.

I feel that the lack of dependency management with good version control in any language is painful and JavaScript was no exception. I recently spent a good part of my day sorting out dependencies in a .NET environment; I sorely missed Apache Maven there. Given my JavaScript development activities I decided to provide the dependency management functionality of Maven to JavaScript via a Maven Plugin.

Imagine writing just the following in your JavaScript code when you want to ensure that jQuery or Prototype.js is present at runtime:

var $;

Alternatively when there are no global variables declared by a JavaScript dependency you can import it without being concerned about its file location and version:

/**
 * @import com.jqueryui:jquery-ui
 */

The com.jqueryui:jquery-ui artifact along with its version is declared in Maven's POM file.

This frictionless approach to declaring JavaScript dependency requirements is the motivation for the Maven JavaScript Import Plugin.

I'm pleased to announce the availability of my JavaScript Import Plugin at Codehaus where I have now also become a committer (a great honour!).

Along the way I decided to create a plugin that provided efficient JSLint invocation during JavaScript development. This plug is also released at the Codehaus and is named the JSLint Plugin.

Please help the professional JavaScript developer community by downloading and building the projects from source and try out some development using the plugins.

Maven crashing with a bus error/invalid memory access given Java 10.6 update 3

2010-11-04T11:18:00.001+11:00

I had this very annoying scenario where all of a sudden the following happened:

mvn

Invalid memory access of location 0x133cd800a rip=0x100504444

Bus error

This seemed to happen as a result of updating to Java 10.6 update 3 on Mac OS X Snow Leopard; but not straight away!

After a while I started to wonder if the update was installed correctly. So, I manually downloaded the update from here:

http://support.apple.com/kb/dl972

...and presto, things appear to be ok again... let's see how long that lasts!

I understand that as of this update, Apple are handing over control to Oracle in terms of managing the distribution of Java on Mac OS X. I think that this is generally good given that Apple has always been a little behind in terms of the latest Java release. Let's hope that the quality of the distro on Mac OS X does not go south though!

When is a unit test not a unit test?

2010-10-06T10:39:00.001+11:00

There's a great deal written on this topic already, but obviously not enough. I had a very interesting chat with a colleague the other day about what constitutes a unit test vs integration, system etc.

I think the boundaries are mirky, but for me at least, a unit test is something that can be constructed with little friction and it can execute fast. I don't really care much about whether I'm testing things that are technically outside of the "unit" I'm working on e.g. if there's an in-memory database available and I can easily construct a test of my code, I don't think I'm crossing the boundary into integration testing. In fact what I've found is that it is sometimes easier to set up an in memory database for the purposes of testing rather than mocking it.

Now of course when you start crossing process boundaries you are moving into the world of integration tests. This falls into the category of "being slow to run" though.

At the end of the day I'm not too worried about whether my test is a unit test so long as its easy to write and quick to execute. If it isn't then I'm happy to have it fall into another category of testing.

A MessageBodyWriter isWriteable method for a collection example

2010-08-07T17:24:00.001+10:00

I can’t believe how much time I’ve spent today on providing an isWritable method for my JAX-RS MessageBodyWriter class. I just couldn't find a good example out there that did what I wanted to do. Maybe Google search wasn't behaving today!

My goal was to have a provider that translates a List<GPSTrackerCollection> object into an application/xml output stream. GPSTrackerCollection is my own class.

The tricky thing was learning how to test for the presence of a List of GPSTrackerCollection objects within my provider.

First things first, I needed to wrap my collection in a GenericEntity object in order to preserve information on the parameterised type; the runtime strips this information of course (this is known as erasure). Here's an example service implementation from my code base:

@GET
@Path("/history")
public Response getHistory() {
  List<GPSTrackerCollection> list = new ArrayList<GPSTrackerCollection>();
  GenericEntity<List<GPSTrackerCollection>> entity =
    new GenericEntity<List<GPSTrackerCollection>>(list) {};
  return Response.ok(entity).build();
}

My MessageBodyWriter ends up looking something like the following:

@Produces("application/xml")
@Provider
public class GPSTrackerCollectionProvider implements
    MessageBodyWriter<List<GPSTrackerCollection>> {

  @Override
  public long getSize(List<GPSTrackerCollection> arg0, Class<?> arg1,
      Type arg2, Annotation[] arg3, MediaType arg4) {
    return -1;
  }

  @Override
  public boolean isWriteable(Class<?> type, Type genericType,
      Annotation[] arg2, MediaType arg3) {

    // Ensure that we're handling only List<GPSTrackerCollection> objects.
    boolean isWritable;
    if (List.class.isAssignableFrom(type)
        && genericType instanceof ParameterizedType) {
      ParameterizedType parameterizedType = (ParameterizedType) genericType;
      Type[] actualTypeArgs = (parameterizedType.getActualTypeArguments());
      isWritable = (actualTypeArgs.length == 1 && actualTypeArgs[0]
          .equals(GPSTrackerCollection.class));
    } else {
      isWritable = false;
    }

    return isWritable;
  }

  @Override
  public void writeTo(List<GPSTrackerCollection> gpsTrackerCollections,
      Class<?> arg1, Type arg2, Annotation[] arg3, MediaType arg4,
      MultivaluedMap<String, Object> arg5, OutputStream os)
      throws IOException {

    ...
  }
}

The key thing here is that we check the parameterised type argument for equality against the class of GPSTrackerCollection. My intuition was that I could do an instanceof operation here, but this is not permitted. Check out this FAQ as to why.

Enjoy.

Application development feelings 2010

2010-07-31T12:45:00.001+10:00

Every now and again I review my practice of developing web based software. Here is a summary of how I write my applications in the context of a multi-tier architecture.

The presentation layer

In essence I'm a big fan of RIA (Rich Internet Application) for situations where I'm interested in developing a web based application as distinct from a web site full of HTML pages. I feel that this is a distinction lost on many people and I also think that those same people struggle with combining HTML/CSS/JS to make a web application.

I'm quite keen on developing something with Sproutcore as in a way, it blows away HTML and CSS and gives you the widgets to build a fully fledged MVC style app that works within the browser; its very similar to developing a native desktop application. The same can be said for Google Web Toolkit (GWT) however I slightly favour Sproutcore as it embraces JavaScript (JS) as a language whereas GWT embraces Java that produces JS i.e. Sproutcore "feels" lighter. Perhaps I should get over that... what Google have developed in the form of Google Wave is impressive.

There are other JS MVC frameworks out there (JavascriptMVC and PureMVC for starters), but I'm not sure that they drop HTML/CSS as a dependency for the developer to consider. I suppose what I'm feeling is that HTML and CSS do not belong in a web based application. Given HTML's start in life as a mark up language for documents I'm comfortable in acknowledging that it does not have a place in my web application.

The biggest thing for me right now with regards to JS RIA is the lack of development tooling. I've been using the Jetbrains WebStorm IDE for a couple months and while I'm impressed on its Window's implementation, it doesn't feel like a native Mac OS X application. Given that Mac OS X is where I like to spend a lot of my time this unfortunately creates a problem for me.

In my ideal world I'd be using Apache Maven for building and deploying my JS RIA's. I've really come to appreciate Maven in the Java space and it is just waiting for someone (may be even me!) to extend it for JS development (I'm familiar with JS Maven Tools but I understand that this project hasn't had activity for a long time). The pressures of family life and my daily job curtails the development of developer tools.

I shall pursue the use of SproutCore for my next RIA. So far I've just used HTML/CSS/JS and jQuery having implemented MVC in the raw. However when it comes to developing a website with perhaps a few dynamic requirements, may be I'll stay with trusty old JSP/Struts that communicates with my logic layer. When it comes to mobile and desktop, I will continue to do as the Romans do and develop native applications that communicate with my logic layer.

The logic layer

I've used Apache Camel on several projects now and just love it. What I end up with is a Java based application that can be managed as a service typically deployed to some brand of Unix. I've been using VMware's Hyperic to monitor the applications and given JMX I get some very deep monitoring capabilities.

I just love the Spring IoC framework and associated toolkits; particularly Spring JPA DAOs and Spring Test.

My applications tend to consist to lots of components that can exist independently of each other, and each with minimal dependencies on other frameworks and toolkits. For example, my application services containing the important business logic tend to be written as Java beans and have no or very little dependence on external libraries.

I lean towards creating RESTful services and look to leverage Apache CXF in this regard. I'm not yet using a toolkit to scaffold some of my RESTful DAO functionality but will at some point. I'm also a big fan of JSON as a payload format but XML's schema definitions have an important place in my heart. JSON with schemas would be a killer and I've just got to get into this.

Messaging is key and Active/MQ has been a great performer. I'm really looking forward to v.5.4 and its support of HTML 5 web sockets. This will allow messaging to be utilised by the JS RIA and I find that prospect quite exciting.

Apache Maven plays a huge role in developing and deploying my logic layer applications.

The data layer

I've been very happy with Postgres as a database and look forward to its forthcoming redundancy features. I'm also curious about Big table style approaches and Document Oriented Databases. I'm not quite sure why SQL should continue to play an application development standpoint (reporting, data mining etc. yes, but perhaps not within the application itself). ORMs have pushed me to the "NoSQL" conclusion and I like not having to deal with queries as much.

Summary

Multi-tier works for me. RIA appears to be the way of the future for web applications as distinct from web documents/pages. If you've not seen Apache Camel check it out. Postgres is great but I'm curious about Document Oriented Databases.

Please share your thoughts.

Development using Salesforce

2010-07-30T08:10:00.001+10:00

I'm presently on a short contract developing using the Salesforce platform.

I've been using Salesforce for over 2 months now and I'm afraid to conclude that it doesn't make it as a viable platform for cloud application deployment. The reason for this is quite simple: it forces you to develop in the cloud as distinct from deploying to the cloud.

Salesforce thought that it would be great to re-invent the wheel, introduce a new language (Apex) and introduce the associated tooling. From a developer perspective this means that all of your regular tooling such as version control, continuos integration, deployment management, dependency management... I could go on... simply isn't there. Let me re-state a bit of that: no version control!

Developing on the Salesforce platform also means that you're dealing with 300ms (ish) round trips for every resource in their web development screens. This is because the platform is hosted in the US and the speed of light being what it is. A developer can quite commonly be waiting for several seconds while the development web app is doing its thing; reminds of my COBOL days 25 years ago when we had to submit stuff to a shared compiler resource and wait.

I'm quite turned on by the idea of deploying to the cloud and I think that VMforce and Google App Engine have chosen the right approach; they of course allow you to develop locally and then deploy to the cloud. Full scale application development in the cloud though is another thing and frankly Salesforce have got it wrong.

Invoking Maven and having it ignore your local settings

2010-06-04T09:31:00.001+10:00

I recently found myself working remotely from my network and needed to perform a build using Maven. My ~/.m2/settings.xml file is configured with information about my network's Nexus repository and a few other things. I therefore needed to find a way to tell Maven to ignore my regular settings; preferably without mutating my settings.xml file. In particular I just wanted to have Maven pull out a plugin from the Central repo.

The documentation on the Maven command doesn't mention how it can be invoked to ignore your settings.xml. However if you type the following then it will ignore it (in my case I wanted to download the wonderful Cobertura plugin and produce a code coverage report) :

mvn -s/dev/null cobertura:cobertura

All good.

Flash on the iPhone

2010-05-12T11:35:00.001+10:00

Given Steve Job's most recent letter concerning Flash on the iPhone, I'd like to broaden the debate to native plugins on the iPhone. Mr. Jobs provides a host of reasons why Flash is not permitted and it reminds me largely of his previous argument around needing native applications on the iPhone (remember that they encouraged us to create nice web apps instead at first; then there was the whole multi-tasking debate..).

There reaches a point with web applications where HTML, Javascript and CSS just can't do what you need them to do. These technologies have come an awful long way and HTML5 in particular is great. I spend a lot of time with these technologies and they are top of mind when considering new applications.

Let's take a concrete example of when you do require a plugin: Google Earth. Google Earth exists as a web plugin as well as a standalone executable. There are times when you'd like to embed Google Earth in a web page given its 3D capabilities; that's why the plugin exists of course!

Another example is with our own web plugin; it is something similar to Google Earth and has advanced capabilities with regards to map projections. We rely on third party C++ libraries and we utilise OpenGL.

Actually I think when it comes to 3D and the use of, say, OpenGL, web technologies fall short. There is a Mozilla proposal to expose OpenGL ES to Javascript, and that's great, but ES doesn't provide full OpenGL capability. Believe me, there are times when you need full OpenGL capability.

If I thought about it I'm sure there are other applications with dependencies such that you need to go native with a web plugin; not often perhaps, but we need to leave the native plugin door open for these situations.

I think Mr. Jobs has his reasons for not supporting Flash, but that's a separate consideration to supporting native web plugins.

In the end, I think that Adobe and Apple will come to an agreement and Adobe will release a well written and performant Flash plugin. The rest of us will then be able to install and write plugins for the iPhone when they are required.

Spring PropertyEditor for JSON map and array properties

2010-04-16T12:43:00.002+10:00

I'm a big fan of Spring but found that it does not support map and list expressions in a property file to be substituted when using a PropertyPlaceholder. So I rolled my own PropertyEditor and can now express a map using JSON in my properties file e.g.:

myMapProperty ={"en-AU": "SKY", "ms": "SKY"}

Here's the property editor (depends on json-lib):

package <packagename>;import java.beans.PropertyEditorSupport; 
import net.sf.json.JSON;
import net.sf.json.JSONSerializer;
/**
 * Converts JSON expressions to Java arrays or maps and vice versa.
 *
 * @author huntc
 *
 */
public class JSONPropertyEditor extends PropertyEditorSupport {
@Override
  public String getAsText() {
    Object object = getValue();
    JSON jsonObject = JSONSerializer.toJSON(object);
    return jsonObject.toString();
  }
@Override
  public void setAsText(String text) {
    JSON json = JSONSerializer.toJSON(text);
    setValue(json);
  }
}

and here's what you must declare in your spring configuration xml:

<bean id="customEditorConfigurer"
  class="org.springframework.beans.factory.config.CustomEditorConfigurer">
  <property name="customEditors">
    <map>
      <entry key="java.util.Map">
        <bean class="<packagename>.JSONPropertyEditor" />
      </entry>
    </map>
  </property>
</bean>

The above indicates that for any required substitution of a map, use my property editor e.g.:

<bean id="someBeanWithAMapProperty"
  class="someClass">
  <property name="someMapProperty"
    value="${myMapProperty}" />
</bean>

iPhone and multitasking

2010-04-13T11:40:00.003+10:00

So here we are. Apple appears to be finally opening up the threading APIs to developers on the iPhone.

I've had numerous conversations with colleagues since the iPhone came out and they've been all sorts of reactions as to why there's no multithreading. Actually, now that I've just typed multithreading let's get this straight. Multitasking relates to the human experience of doing multiple things at the same time; it is a use-case; it is something that I've been told (by my wife) that men can't do and women can (!).

Multithreading is the ability for an OS to cooperatively or pre-emptively (generally the latter) slice processor time between different paths of execution within and outside of a number of processes (generally applications).

The iPhone has supported multitasking and multithreading forever. The difference now is that Apple appears to have opened up the multithreading API to non-Apple developers. I've not seen the details on this, perhaps they've constrained the thread APIs in some interesting way with the goal of limiting battery usage, but it is a very good (and necessary) thing that there will now be multithreading.

One application that I'm really looking forward to using is Skype. Finally, we'll be able to receive Skype calls with the same experience of receiving regular mobile calls; so long as you're on a WiFi network of course. I think that this will be amazing as I'm never near my computer when someone wants to contact me via Skype. However, I always have my iPhone at hand of course.

I can't wait for SIP supporting applications to come along for the iPhone either. The prospect of receiving my home phone calls when away from home via VOIP is fantastic.

Oh and one thing I'd like to joyfully throw back at the multithreading doubters: get over it. Yes, multithreaded programming is harder but its just the way the world is. Multiple things happen at the same time. Embrace it.

Well done Apple. I look forward to iPhone 4.0.

The pitfall of setting a list of parameters on a JQL query

2010-04-12T14:31:00.007+10:00

I recently had a need to provide a collection as a parameter to a JQL IN expression. This is what I did:

List results = getJpaTemplate()
  .find(
    "from AbstractEvent e "
    + "where "
    + "(e.subscription, e.eventTime) in "
    + "(select e.subscription, max(e.eventTime) "
    + "from e "
    + "where "
    + "e.class in "
    + "(SentSMSFlightSubscribed,"
    + " ReceivedSMSFlightUnsubscribe) and "
    + "e.subscription.flightNo in (?) and "
    + "e.subscription.scheduled = ? "
    + "group by e.subscription)",
    renderedFlight.getFlights(),
    renderedFlight.getScheduled());

The problem with the query is that you get a class cast exception when an attempting to substitute the collection (renderedFlight.getFlights) .

To fix this you simply need to be explicit about the parameters:

    + "e.subscription.flightNo in (?1) and "
    + "e.subscription.scheduled = ?2 "

Problem solved.

Updated: Sunday 11 July 2010: Even using position parameters doesn't always work. While it works in the above example it didn't work for me when using a parameter of a non-primitive type; at least I think that was the issue. So in a nutshell, if the above doesn't work for you then just use findByNamedParams instead.

You may also note this beauty of a line:

e.class in (
 SentSMSFlightSubscribed,
 ReceivedSMSFlightUnsubscribe)

That is how you perform an "instanceof" style of operation in JQL. Nice.

Java's Swing framework for Javascript and Canvas

2010-02-28T00:51:00.004+11:00

It is 0051 on Sunday morning and I just had a wild thought...

I've been developing applications in Javascript using jQuery and taking an interest in Sproutcore. I love jQuery but most definitely subscribe to Sproutcore's architecture where it makes a very strong statement about web applications being distinct from web documents (there is a place for both).

...and then it occurred to me... why not try and implement Java's Swing framework in Javascript and use the Canvas element almost exclusively for web applications? Do away as much as possible with CSS and HTML - I don't think they're great for web applications anyhow...

I don't doubt that it'd be a lot of work implementing Swing in Javascript, but there's an awful lot of design to be leveraged there.

I think I'll sleep on it...

Does Grails lower the barriers of entry to programming?

2010-02-17T21:08:00.005+11:00

Grails is the Groovy equivalent of Ruby on Rails and I've been working with Grails for the past couple of months.

I'm a big fan of programming by convention and adopted Apache Maven many moons ago for just this reason. Grails also promotes programming by convention and by and large its quite reasonable for building server hosted web applications.

However one thing in particular has been bothering me. It seems that there is a view that Grails saves you the trouble of worrying about what goes on underneath it. This attitude is similar to one where using ORMs such as Hibernate save you the need to understand databases.

These attitudes are fraught with dangers as has been beautifully pointed out by Joel Spolsky in his Leaky Abstractions article.

I would agree that Grails can assist in increasing productivity but not at the expense of knowing what goes on under the hood. As Joel puts it, "... the abstractions save us time working, but they don't save us time learning".

Today I came across a Grails situation where I compared a newly instantiated domain object with one obtained through GORM. From my perspective both objects were equivalent as they had the same identifiers. I had wrongly assumed though that GORM had adopted the Hibernate convention of providing hashCode() and equals() on a natural key. Because Grails does all this magic for me, I had just assumed that this would be done. Well, it doesn't; and that's fair enough; how would GORM know what the natural key is? (declaring the natural key could be a nice feature for GORM). The point is that I could quickly find the problem because I'm familiar with Java/JPA/Hibernate and know that hashCode() and equals() should be provided for natural keys.

My advice is to stay curious. Don't be content with copy and pasting a bit of code you found using Google, seeing it work but not understanding it. This situation will bite you on the bum when your product goes live and you get a problem that you can't diagnose because you don't really understand what the code is doing.

Programming computers is a complex business and that's a fact that should not be avoided. Embrace the complexity.

Processing Javamail emails for specific mime types

2010-02-13T22:18:00.004+11:00

I recently had to write some software that received an email and processed it for iCal events. I wanted my program to be tolerant of receiving the iCal events within the body or as a attachment. Furthermore if the attachment was a mail message then I wanted to look inside that email for the event and so on...

When I thought about the problem initially I thought that processing just the attachments of an email would be sufficient. That's not the case given Microsoft Outlook though. I quickly learnt that Outlook sends its iCal events within the body of an email and not as an attachment.

I ended up with the following bit of Javamail/iCal4j/Java code that should generally be useful for processing mail messages. In my case I'm looking for a text/calendar object but it could be anything of course. The addCalendarDataToCalendars function can be called recursively and adds any calendar objects it finds to the collection passed in. Enjoy.


  /**
   * Process a data handler's content looking for a 
   * calendar object.
   * 
   * @param calendars
   *            the collection of calendars that we 
   *            must add any found calendar to.
   * @param dataHandler
   *            the data handler to use.
   */
  private void addCalendarDataToCalendars(
      Collection calendars,
      DataHandler dataHandler) {
    MimeType mimeType;
    try {
      mimeType = 
        new MimeType(dataHandler.getContentType());

      if (mimeType.getBaseType()
            .equals("text/calendar")) {
        // Calendar data is what we're ultimately 
        // searching for.
        CalendarBuilder builder = 
          new CalendarBuilder();
        Calendar calendar = builder.build(
          dataHandler.getInputStream());
        calendars.add(calendar);

      } else if (mimeType.getBaseType()
                   .equals("message/rfc822")) {
        // Found an object representing an email 
        // address in its entirety.
        Message mailMessage = (Message) dataHandler
            .getContent();
        addCalendarDataToCalendars(calendars, 
          mailMessage.getDataHandler());

      } else if (mimeType.getBaseType()
                   .startsWith("multipart/")) {
        // Found a bunch of attachments so 
        // process them each.
        Multipart multipart = 
          (Multipart) dataHandler.getContent();
        for (int i = 0, n = multipart.getCount(); 
             i < n; ++i) {
          Part part = multipart.getBodyPart(i);

          String disposition = 
            part.getDisposition();

          if ((disposition != null)
             && ((disposition.equals(Part.ATTACHMENT) 
             || (disposition.equals(Part.INLINE))))) {
            addCalendarDataToCalendars(calendars, 
              part.getDataHandler());

          } else if (disposition == null) {
            MimeBodyPart mimeBodyPart = 
              (MimeBodyPart) part;
            addCalendarDataToCalendars(calendars, 
              mimeBodyPart.getDataHandler());

          }
        }
      }

    } catch (MimeTypeParseException e) {
      logger
          .warn("Ignoring attachment - could not " +
                "parse the mime type of: "
              + dataHandler.getContentType()
              + " - " + e);
    } catch (IOException e) {
      logger.error(e);
    } catch (ParserException e) {
      logger
          .warn("Ignoring attachment - could not " +
                "interpret the calendar object: "
              + dataHandler.getName() + " - " + e);
    } catch (MessagingException e) {
      logger
          .warn("Ignoring attachment - could not " +
              "decode the calendar object: "
              + MimeUtility.getEncoding(dataHandler) 
              + " - " + e);
    }
  }

Eclipse and friends are memory hungry

2009-12-28T12:37:00.003+11:00

I've been using Eclipse Galileo for some time now and have quite a few favourite (and necessary!) plugins installed. These included m2eclipse, subeclipse, findbugs, checkstyle and many more.

I had noticed though that things started to slow down. Even clicking between tabs was slow (started out ok after launching and then got worse over time).

This smelled like a memory performance issue so I re-configured Eclipse to have a maximum memory size of 2GB and all now appears well. 2GB may have been an over-reaction - your mileage might vary, but for me it seems to have done the trick.

Chrome on Mac - No Google Earth plugin though

2009-12-10T11:01:00.008+11:00

I've just installed the Chrome beta on my Mac. Looks good overall.

However the first thing I did ('cause its close to what I'm involved in) was try out the Google Earth Plugin.

No dice:

So perhaps the same constraints holding back the plugin under Safari 4 are holding up it working under Chrome also; perhaps it is the out of process plugin architecture, and/or the new requirement to use Core Animation to render OpenGL...

Actually I've noticed that Chrome doesn't presently support the Core Animation for OpenGL NPAPI plugins (my CA NPAPI plugin isn't able to instantiate - most likely due to the browser stating that CA/Cocoa is unsupported due to the graceful failure).

In searching for some technical insight I noticed that Google provide a great article on the motivation behind Chrome.

First impressions then are that Chrome looks interesting but it isn't quite there for me yet given my specialised NPAPI requirements.

MacPorts - my new best friend

2009-12-04T16:11:00.005+11:00

Perhaps it is just that I'm getting older, but I've resisted using package managers on Linux and Mac OS X for some time. I dunno, I guess I preferred to download the source and build it myself. What a sucker I am for punishment!

I think that I had a similar resistance to Apache Maven originally - I just didn't want the learning curve of a new tool.

I'd got to the point with a C++ project that I was fed up managing, building and sorting out the dependencies of various C/C++ based libraries. The catalyst for me was when I started using Cairo, a 2D vector and imaging library (which is great!). Cairo had many dependencies and the project recommended a number of package managers for my OS X build environment.

I ended up with MacPorts and haven't looked back. Managing those dependencies just got a whole lot easier. MacPorts was apparently started by a group of Apple engineers and it grew from there. At the time of writing there are now over 6,000 packages available through MacPorts.

MacPorts likes to put everything in /opt by default. For some bizarre reason I was against using /opt as my root folder for all MacPorts derived packages - talk about resisting change for no good reason! However I got over that and must take a leaf out here and try to resist such time consuming temptations. Who really cares where MacPorts puts stuff so long as at the end of the day your dependencies are being managed and your applications build?

The only thing I found myself changing with the default configuration was the /opt/local/etc/macports/variants.conf file. I wanted +universal as a default. My projects tend to build both 32 bit and 64 bit targets and so I needed MacPorts to do this by default (I kept forgetting to specify the variant on the command line).

All good. MacPorts is great and is another tool that helps me focus on the problem at hand.

Write less code

2009-11-26T11:58:00.004+11:00

I think that old adage of "Sorry I wrote you such a long letter, if I had more time I'd write you a shorter one" certainly applies to software development as well. If you spend a great deal of time learning how to write less code by using more off-the-shelf toolkits and frameworks you will write less code in the end. Sounds obvious, but I still see developers re-inventing the wheel far too often.

The following code requires a steep learning curve around Java, Apache projects, iCal4j (a great calendaring toolkit for iCalendar) Spring, Spring DAO and Maven; and probably many more things. Look how straightforward the expression is though:


from(calendarEventEmailInboundEndpoint).to(
  "bean:calendarAttachmentTranslator").to(
  "bean:calendarEventTranslator").to(
  "bean:calendarEventDAO?method=save");

The above Java code shows a Camel route that reads emails from a pop3 or imap inbox (depending on how calendarEventEmailInboundEndpoint is configured), extracts iCalendar attachments from one or more emails, translates the iCal4j representation into a model form for persistence, and finally persists the calendar event(s) in a database.

Pretty sophisticated stuff for 4 lines of code. :-)

Maven Release Plugin - My new friend

2009-10-18T20:16:00.008+11:00

One tool that slipped my radar until only this week was the Maven Release Plugin. If you are a Maven user and you release what you produce with it (I would hope so!) then you should take a look at this plugin.

My applications tend to be containers for the many pre-fabricated components of other projects I have. For instance I have 19 projects in my Eclipse workspace where 3 of them are the containers - if you like, the applications themselves.

Having lots of projects means that it can be easy to miss updating the version numbers for them when it comes time to perform a release; particularly missing that a project is being depended on as a snapshot.

In summary two big reasons to use the Maven Release Plugin are:

it ensures that there are no snapshot dependencies; and

it updates your pom's version number automatically.

In order to use the plugin I found myself having to add an scm element, an element for the plugin itself and a distributionManagement element.

The scm element let's the plugin know where to tag a release. Each time you make a release the plugin will create a tag for it in your source code repository. I was doing this as a manual task prior to the plugin so this feature is great.

I configured the plugin to ignore deploying my source code and javadoc to the repository. I actually had a problem with the maven source plugin given spaces contained in my path. I also didn't need to deploy the source code as I have it within my scm repository and I don't intend releasing the source to other parties.

Lastly you need to let the deploy plugin know how to deploy your project's artifacts in a maven repository i.e. the jar files in my case. The release plugin will invoke the deploy goal by default (actually it does a deploy and then "site deploy"). My projects generally rely upon the default goal with exception to those "application" projects. The application projects simply invoke the assembly plugin as I'm interested in producing a tar.gz file of the entire application. I therefore don't need to specify repository deployment details in my pom for these application projects as there is none.

All in all the maven release plugin saves me time and improves my release build quality.

Client/server or consumer/provider

2009-09-17T21:06:00.010+10:00

Wow, I can't believe that it is now September and I've not posted anything since July. In a nutshell I landed a new airport customer (Subang Skypark - the Kuala Lumpur domestic airport) and so life has been terribly busy. As a result there's a lot of software development to be discussed since my last post. However I'll start with this little one as a result of an interesting conversation today.

I designed an interface today that describes a set of remote services. As the interface is a consumer of those services I came up with the name XXXConsumer (I didn't really use XXX though). A friend of mine asked why not call it XXXClient? I had to really think about that.

Two seconds later I blurted out that client and server had become an overloaded term and that the hip terms are now consumer and provider respectively. In other words I didn't really know why I preferred to use the word consumer and provider and put it down to the time I've spent with JMS and Apache Camel. These technologies often use the terms consumer and provider.

Wikipedia offer a nice definition of client and server that effectively implies that these names are used for fairly course-grained objects such as applications and systems. My assertion having thought about this a little more is that consumers and providers are used in a finer-grained context. By finer-grained I refer to the software components found within my application.

In addition client and server are certainly overloaded terms and typically imply that an application or system has just this one role. By contrast I often see that components are both consumers and providers of services. Perhaps this is another difference.

Thank you for allowing me to client^H^H^H^H^Hconsume your time!

Replacing jsvc for Java daemons

2009-07-17T15:00:00.005+10:00

For some time I have been using the Apache Commons Daemon project to run and maintain my Java based services. jsvc did/does a reasonable job but I start to worry when a project has not been touched for a while.

In addition I am in the midst of deploying a Java based service on Mac OS X and so I am in the world of launchd. I wanted a better way of controlling Java based services; particularly one that did not fork multiple processes given launchd's garbage collection.

The principal issue with a Java service running in the background is that you need to have it respond asynchronously to various singals; particularly SIGTERM which of course is issued when the OS is being shutdown. My service needs to shut down gracefully.

Enter in a feature of Java 1.4.2 that I did not realise existed: Shutdown Hooks. In essence Shutdown Hooks provides your Java application an opportunity to respond to the application quitting.

Followers of this blog know that I am an Apache Camel addict. You will not therefore be surprised to find that what follows is an example of how to start up and shutdown a Camel context using Java's Shutdown Hooks.


public class EntryPoint {
  static Logger logger = 
    Logger.getLogger(EntryPoint.class.getName());

  static EntryPoint entryPoint;

  Main main;

  public static void main(String[] args) {
    entryPoint = new EntryPoint();

    Runtime.getRuntime()
     .addShutdownHook(new Thread() {
      public void run() {
        try {
          entryPoint.stop();
        } catch (Exception e) {
          logger.fatal(e.toString());
        }
      }
    });

    try {
      entryPoint.start();
    } catch (Exception e) {
      logger.fatal(e.toString());
    }

  }

  public void start() throws Exception {
    logger.info("Starting up");

    // Start up the context
    main = new Main();
    main.start();

    logger.info("Started");
  }

  public void stop() throws Exception {
    logger.info("Stopping");

    // Shutdown the context
    main.stop();

    logger.info("Stopped");
  }
}

A Camel based XML Payload HTTP polling provider

2009-07-10T17:19:00.008+10:00

Wow, what a mouthful of a title that is.

The EAI Polling Consumer pattern is well documented. Polling consumers are particularly useful for HTTP clients such as AJAX applications. Their presence provides a means of implementing publish and subscribe.

What I needed was some code to service my AJAX consumer; a Polling Provider so to speak.

Given my immersion in Apache Camel I came up with a Polling Provider for XML objects provided by XMLBeans. The provider assumes that an HTTP endpoint is to be interacted with and thus sets the HTTP time headers appropriately.

The Polling Provider holds the notion that the xml object is updated by one thread and then consumed by another. Furthermore the time the xml object was last updated is retained. This allows HTTP consumers to specify the HTTP ifModifiedSince header and block if the condition is not met.

To update the provider (fidsDocumentPollingProvider is an instance of it and newFidsDocument is an XML document object):


fidsDocumentPollingProvider.update(
  newFidsDocument, new Date());

To retrieve the resource in a RESTful manner:


// Provide a RESTful service to retrieve FIDS data
from("jetty:http://0.0.0.0:9000/FIDSService/FIDS")
  .inOut("direct:getFIDS");

// Provide a component to process FIDS requests
from("direct:getFIDS")
  .process(fidsDocumentPollingProvider);

Here is the code I came up with for the provider. I would be very interested to hear of better ways to do this given Apache Camel.


package com.classactionpl.camel.xmlbeans;

import java.text.ParseException;
import java.text.SimpleDateFormat;
import java.util.Date;
import java.util.Locale;
import java.util.TimeZone;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.locks.Condition;
import java.util.concurrent.locks.Lock;
import java.util.concurrent.locks.ReentrantLock;

import org.apache.camel.Exchange;
import org.apache.camel.Processor;
import org.apache.camel.component.http.HttpProducer;
import org.apache.xmlbeans.XmlObject;

public class XMLHttpPayloadPollingProvider implements Processor {
  XmlObject xmlObject;
  Date lastModified;
  Lock lock = new ReentrantLock();
  Condition update = lock.newCondition();

  public void update(XmlObject xmlObject, Date lastModified) {
    lock.lock();
    try {
      this.xmlObject = xmlObject;
      // We don't need the milliseconds and they can upset
      // our comparisons given that they are not passed from the
      // outside world.
      this.lastModified = new Date((lastModified.getTime() / 1000) * 1000);
      update.signal();
    } finally {
      lock.unlock();
    }
  }

  public void process(Exchange exchange) throws Exception {
    lock.lock();
    try {
      boolean waitForUpdate = true;
      Date ifModifiedSince = null;
      if (lastModified != null) {
        String ifModifiedSinceStr = exchange.getIn().getHeader(
            "If-Modified-Since", String.class);
        if (ifModifiedSinceStr != null) {
          try {
            SimpleDateFormat rfc822DateFormat = new SimpleDateFormat(
                "EEE', 'dd' 'MMM' 'yyyy' 'HH:mm:ss' 'Z",
                Locale.US);
            ifModifiedSince = rfc822DateFormat
                .parse(ifModifiedSinceStr);
            waitForUpdate = (lastModified
                .compareTo(ifModifiedSince) <= 0);
          } catch (ParseException e) {
          }
        }
      }

      boolean provideDateHeader;

      if (waitForUpdate) {
        update.await(90, TimeUnit.SECONDS);
      }

      if (xmlObject != null) {
        if (ifModifiedSince == null) {
          ifModifiedSince = lastModified;
        }
        if (lastModified.compareTo(ifModifiedSince) > 0) {
          exchange.getOut().setBody(xmlObject.newInputStream());
          exchange.getOut().setHeader("Content-Type", "text/xml");
          provideDateHeader = true;
        } else {
          exchange.getOut().setHeader(
              HttpProducer.HTTP_RESPONSE_CODE, 304);
          provideDateHeader = true;
        }
      } else {
        exchange.getOut().setHeader(HttpProducer.HTTP_RESPONSE_CODE,
            404);
        provideDateHeader = false;
      }

      if (provideDateHeader && lastModified != null) {
        SimpleDateFormat rfc822DateFormat = new SimpleDateFormat(
            "EEE', 'dd' 'MMM' 'yyyy' 'HH:mm:ss' GMT'", Locale.US);
        rfc822DateFormat.setTimeZone(TimeZone.getTimeZone("GMT"));
        exchange.getOut().setHeader("Last-Modified",
            rfc822DateFormat.format(lastModified));

      }

    } finally {
      lock.unlock();
    }
  }
}

Titan Class goes into production

2009-05-14T20:06:00.004+10:00

She's finished, well as finished as they always are - which of course means that they are never finished.

I am referring to my work over the past few months which is really just a concerted effort building on years of work. I'm now in a position to track vehicles, containers, you name it.

From a software development perspective there are many things that I'm happy about. I've previously discussed jQuery which really has eased the AJAX programming. From a backend perspective I am using the amazingly functional, scalable and distributable Apache Camel (also discussed in other posts).

One thing with the client-side that has really impressed me is the Google Earth plugin. I hope that you agree that it works very, very well (a link to the site is provided at the end of this post). Some of our potential customers will require their own maps along with their own projections and Google Earth will not handle that (Antarctica for example). However for many scenarios, Google Earth will be great.

Incidentally where we will be required to use specialised maps with funky projections, I have written a Java applet using JavaFX. The applet supports many types of map projections and is based on the work of Proj.4 and friends.

Here is a quick list of the OSS projects I utilised (and even contributed back to in some cases!):

Apache ActiveMQ,
Apache Camel
Apache Directory Server
Apache MINA
Apache Tomcat
Apache Web Server
GEOS
Eclipse
Hibernate
Hibernate Spatial
jQuery
JSON
JTS
JUnit
Linux
Log4J
Maven
PostgresQL
PostGIS
Proj.4
Spring
Subversion
xmlbeans

Go OSS!

Feel free to visit the tracking site and play around with the demo data.

Guice

2009-05-11T09:45:00.004+10:00

I found this video introduction on Guice very good. Guice is another Inversion of Control (IoC) container, much like Spring IoC, but with the advantage of coming later. What's particularly interesting about Guice is that it largely forms the basis of JSR-299's IoC.

I'm a big user of Spring IoC and frankly don't have a problem with it. However I can see that Guice could save me some wiring code and, therefore, potentially bugs (not that there are bugs in my code!). I might well consider using Guice for future projects. The outcome of JSR-299 will weigh in heavily on this decision though as I like my code to be as standards-compliant as possible.

One thing that I am comfortable about with Spring though (vs Guice) is that my beans have no knowledge of being injectable; they simply are by being bean conforming. With Guice you must declare what is potentially injectable. I do like the loose coupling between my bean code and the IoC that Spring provides, yet I can see that Guice, by declaring what is injectable, can save some of the wiring code.

I'll guess I'll just have to give Guice a whirl to find out.

jQuery

2009-05-06T11:18:00.012+10:00

I have just completed an Ajax application that uses the jQuery framework. jQuery's promise is to "write less, do more", which I suppose is the objective of most frameworks. Does jQuery deliver though? In a nutshell, yes.

Writing an Ajax application is hard; not because of network paradigms or Javascript, but because each browser brings with it some quirk. Catering for all of these browsers is simply time consuming so, by hard, I really mean time consuming. It is therefore hard to knock out an Ajax application quickly given the shear amount of cross-browser testing that has to be performed.

jQuery does not eliminate cross-browser issues but it certainly minimises them. With my application, the only real short-fall I found with jQuery was its lack of support for dealing with XML documents (not XHTML). This was primarily due to the lack of XML namespace support and I think that this is an area that jQuery should focus on. However it is important to note that the real culprit for the lack of namespace support is Internet Explorer. In essence there is no support for namespaces with IE. I find this extremely difficult to understand... ok, IE 6 with no namespaces is understandable to a degree, but version 7, and worse even version 8 do not support XML namespaces. Microsoft virtually invented SOAP given their backing of it, yet their browsers can't parse SOAP documents accurately (you can ignore the namespaces, but then you ignore the value of namespaces of course).

Here are some other things I found myself having to cater for explicitely...

iframe shims

iframes shims are provided by a jQuery plugin named bgiframe and you can set up jQuery's dialog so that it uses an iframe. Unfortunately IE6 is not the only browser that requires iframes for the purposes of overlaying html objects on OS rendered content (buttons, applets, objects etc.). I therefore found myself having to roll my own. Here's a snippet of what needs to be done in the case of displaying a login dialog when some label is clicked:


var loginDialog = $("#loginDialog").dialog({
autoOpen: false,
close: function() {
loginDialogIframeShim.css("visibility", "hidden");
},
...
});
var loginDialogIframeShim = $(document.createElement("iframe"));
loginDialogIframeShim.attr("frameborder", "0");
loginDialogIframeShim.attr("scrolling", "no");
loginDialogIframeShim.attr("allowtransparency", "false");
loginDialogIframeShim.css("position", "absolute");
loginDialogIframeShim.css("visibility", "hidden");
$("body").append(loginDialogIframeShim);

var login = $("#login").click(function() {
loginDialog.dialog('open');
var loginDialogParent = loginDialog.parent();
var offset = loginDialogParent.offset();
loginDialogIframeShim.css("left", offset.left + "px");
loginDialogIframeShim.css("top", offset.top + "px");
loginDialogIframeShim.css("width", loginDialogParent.outerWidth() + "px");
loginDialogIframeShim.css("height", loginDialogParent.outerHeight() + "px");
loginDialogIframeShim.css("visibility", "visible");
...

The above works well for Windows FF and IE 7 and Mac OS FF and Safari. I haven't tested other browsers.

Datepicker ranges

It would be great to see the Datepicker (which is really great as is) enhanced to support a date range selection.

XmlHttpRequest

While jQuery does a lot to abstract away the peculiarities of browsers with XmlHttpRequest, I did find myself having to encode credentials for Safari. This is because Safari likes to send credentials as part of the URL initially (if that fails then it tries the conventional way using an authorisation header - I do not understand why).


if($.browser.safari) {
 requestUsername = encodeURIComponent(requestUsername);
 requestPassword = encodeURIComponent(requestPassword);
 }

$.ajax({
 url: "...",
 dataType: "xml",
 username: requestUsername,
 password: requestPassword,
   ...

Date/time utilities

In general terms I found these lacking. Quite often with XML one has to convert from ISO8601 to a Javascript date and back. I needed to provide these functions:


function formatISO8601Time(time) {
  var timeStr;
  
  var year = time.getUTCFullYear();
  timeStr = "" + year;
  timeStr += "-";
  
  var month = time.getUTCMonth() + 1;
  if (month < 10) {
    month = "0" + month;
  } else {
    month = "" + month;
  }
  timeStr += month;
  timeStr += "-";
  
  var day = time.getUTCDate();
  if (day < 10) {
    day = "0" + day;
  } else {
    day = "" + day;
  }
  timeStr += day;
  timeStr += "T";
  
  var hour = time.getUTCHours();
  if (hour < 10) {
    hour = "0" + hour;
  } else {
    hour = "" + hour;
  }
  timeStr += hour;
  timeStr += ":";
  
  var minute = time.getUTCMinutes();
  if (minute < 10) {
    minute = "0" + minute;
  } else {
    minute = "" + minute;
  }
  timeStr += minute;
  timeStr += ":";
  
  var second = time.getUTCSeconds();
  if (second < 10) {
    second = "0" + second;
  } else {
    second = "" + second;
  }
  timeStr += second;
  timeStr += "Z";
  
  return timeStr;
}

function parseISO8601Time(time) {
  var date = new Date();
  
  var year = time.substr(0, 4);
  var month = time.substr(5, 2);
  var day = time.substr(8, 2);
  var hour = time.substr(11, 2);
  var minute = time.substr(14, 2);
  var second = time.substr(17, 2);
  
  date.setUTCFullYear(year);
  date.setUTCMonth(month - 1);
  date.setUTCDate(day);
  date.setUTCHours(hour);
  date.setUTCMinutes(minute);
  date.setUTCSeconds(second);
  date.setUTCMilliseconds(999);
  
  return date;
}

function formatLocalHoursAndMinutes(time) {
  var timeStr;
  
  var hour = time.getHours();
  if (hour < 10) {
    hour = "0" + hour;
  } else {
    hour = "" + hour;
  }
  timeStr = hour;
  
  var minute = time.getMinutes();
  if (minute < 10) {
    minute = "0" + minute;
  } else {
    minute = "" + minute;
  }
  timeStr += minute;
  
  timeStr += "h";

  return timeStr;
}

XML Namespace utilities

As mentioned before there is very little support namespaces so here's what I had to roll:


function getElementsByTagNameNS(node, namespaceURI, localName) {
    var nodeList;
    
    if (node.getElementsByTagNameNS != null) {
      nodeList = node.getElementsByTagNameNS(namespaceURI, localName);
    } else {
        nodeList = node.getElementsByTagName(resolveNS(namespaceURI) + localName);
    }
    
    return nodeList;
}

function getAttributeNS(node, namespaceURI, localName) {
    var attrVal;
    
    if (node.getAttributeNS != null) {
      attrVal = node.getAttributeNS(namespaceURI, localName);
    } else {
        attrVal = node.getAttribute(resolveNS(namespaceURI) + localName);
    }
    
    return attrVal;
}

function isNode(node, namespaceURI, localName) {
  var nodeMatched;
  if (node.localName != null) {
    nodeMatched = (node.namespaceURI == namespaceURI && node.localName == localName);
  } else {
    nodeMatched = (node.nodeName == (resolveNS(namespaceURI) + localName));
  }
  
  return nodeMatched;
}

function resolveNS(namespaceURI) {
  var namespacePrefix;
  if (namespaceURI == "http://www.yoururigoeshere") {
    namespacePrefix = "ns1:";
  } else if (namespaceURI == "you are getting the idea now hopefully") {
    namespacePrefix = "ns:";
  }
  return namespacePrefix;
}

Summary
All in all though jQuery saved me a lot of code, particularly around manipulating and traversing my XHTML DOM. The long short of this was that my XHTML page is semantically correct containing absolutely no erroneous divs, style or class declarations i.e. no presentation considerations.

My last Ajax application was a couple of years ago and I must say that using a framework like jQuery has certainly improved programming here. I think we are now seeing some maturity in the browsers and the Javascript framework.

I look forward to continued use of jQuery.

Java Persistance with Hibernate - what I had to re-visit after reading the book

2009-03-30T16:58:00.003+11:00

I recently had a JPA related issue and decided that it was high time to purchase a good book on the subject; particularly as JPA/Hibernate tend to play a very important role of course.

I ended up discovering "Java Persistance with Hibernate" (Bauer, King). This book is a revision of "Hibernate in Action" and it certainly reads as a complete reference to all things ORM. What attracted me to this book in particular was its coverage of JPA. I prefer to use JPA over Hibernate interfaces so that my ORM implementation choice is left open as much as possible. Having said that Hibernate gives me no reason to move to another ORM.

I must have been using Hibernate/JPA for the past 18 months. When I started I recall wanting to skim over the ORM subject as much possible in order to get things done i.e. I did not think deeply about the implications of some of my decisions; or rather I misinterpreted the way things worked e.g. the 2nd level cache. To this end here is a list of things that I found myself re-visiting:

* property annotations instead of field annotations;

* using the Collection interface instead of the Set interface;

* cache annotations;

* bi-directional one-to-many relationships;

* immutable entities

Using property access can boost performance when performing operations on a collection (also as opposed to a set). For example if you want to add to the collection Hibernate does not have to load the existing collection from the database. If you use a Set then it does in order to guarantee Set semantics.

Understanding the second level cache is very important and if you don't have time to learn it then disable it. It is also a good idea to understand whether you have immutable entities so that you can use the Hibernate @Entity(mutable=false) annotation and flag to the cache that the object is read only. This permits Hibernate a few optimisations including a minimisation on the number of update statements that require preparing at startup.

I always wondered why my bi-directional relationships were generating a join table. It finally sunk in that the mappedBy attribute on an association describes the foreign key. No more join tables for bi-directional relationships.

There's no excuse; just buy the book if you're doing any ORM. Having said that even before I made these changes Hibernate was performing exceptionally well!

Mutual SSL authentication and LDAP authorisation for ActiveMQ

2009-03-17T22:01:00.011+11:00

I now have quite a lot of software infrastructure supporting the business of tracking things that move; particularly aircraft flights. The back-bone of all of this is ActiveMQ. ActiveMQ is an incredible messaging work-horse and easily handles 300 messages per second being received from a number of radar sensors.

I have also re-developed my Titan Class Vision client application using JavaFX (the subject of another blog entry). The goal of this re-write was to be able to deploy Vision on a wider variety of platforms than I could do so previously. Before the JavaFX version Vision was a turn-key hardware/software combination written in Objective-C/C++ and ran on Mac OS X only.

With the advent of the new version I am able to deploy Vision across the internet. Given this there are suddenly many more potential candidates in terms of users and I had a security question to consider: how can I demonstrate to whoever is concerned that I have made every effort to ensure that this sensitive real-time flight data is not being mis-used by anyone.

Authentication

My requirement therefore became one centered around SSL. ActiveMQ permits connections to be established using SSL. Server only authentication is fairly straightforward and covered here. What I was after though was client certificate verification; otherwise known as mutual SSL authentication. In a nutshell, the server verifies the client's certificate as one it trusts, and the client verifies the server's certificate as one it trusts.

When it comes to the client application, in my case the JavaFX application, you need to make sure that the client's keystore is accessible. Don't do as I did and try using the JRE's default keystore for this purpose. I just couldn't get that to work. Instead do as the ActiveMQ SSL page suggests and provide the client with its own keystore.

Another tip for the client, is to set the javax.net.ssl.* properties within the application itself; before you try establishing the JMS connection of course. I express the location of the client's keystore in relation to the user.home system property.

From a broker (server) perspective, one has to use a JAAS LoginModule that permits certificate based authentication. Fortunately I found an ActiveMQ-JAAS class named CertificateLoginModule (of all things). One very subtle thing to note: when specifying the use of this login module in activemq.xml you must use the jaasCertificateAuthenticationPlugin element instead of the jaasAuthenticationElement. I think that this is because the certificate login module requires a different login callback to obtain the client's certificate.

CertificateLoginModule is only half the picture in the same way that authentication is only half the picture. Authorisation is required and the CertificateLoginModule has to be extended to support this; the login module does not know how to authorise a certificate. I can help there as I have provided this code. More on that later though (I insist on you reading the rest of this entry!).

Finally on authentication, you need to tell ActiveMQ that you want to perform client certificate authentication; it will not do it unless told to do so. You do this by specifying "needClientAuth=true" parameter on the ssl transport in activemq.xml.

You might want to disable the other connectors and open up your firewall just for 61617 SSL connectivity. I think that once you make authentication a priority with the broker then you need to give up the non-secure connectors.

Authorisation

With authentication done (I know who you are, you know who I am), I needed to deal with authorisation (now I know who you are, what am I going to allow you to do). For this I wanted to centralise my user and group/role information. ActiveMQ allows you to specify user/group associations in its configuration file, but I wanted to do what all grown-ups do: specify my users and groups in a centralised LDAP directory.

The CertificateLoginModule requires extension to specify how authorisation is done. I have created a CertificateLoginDirectoryRolesModule that will take the subject DN from each client certificate presented (there can be many but typically just one), and then call upon my LDAP store to determine which groups the DN is a member of.

I have set up my LDAP server (ApacheDS - fabulous) to allow anonymous access but also enabled access controls. This means that, by default, the LDAP server permits very little authorisation with just admin access. I then created a group named "activemq" off the "groups" node and used an ACI to allow anonymous searching of that group. I ended up with a group hierarchy as follows:


ou=system
  ou=groups
    ou=activemq (anonymous users can see this and below)
      cn=jms-services
      cn=activemq-users
      cn=com.classactionpl.javaFlightTopic.Subscribers

Correspondingly here is my authorisation mapping within activemq.xml:


<authorizationPlugin>
  <map>
    <authorizationMap>
      <authorizationEntries>
        <authorizationEntry queue=">"
          read="jms-services"
          write="jms-services"
          admin="jms-services" />

        <authorizationEntry topic=">"
          read="jms-services"
          write="jms-services"
          admin="jms-services" />

        <authorizationEntry topic="com.classactionpl.javaFlightTopic"
          read="com.classactionpl.javaFlightTopic.Subscribers" />

        <authorizationEntry topic="ActiveMQ.Advisory.>"
          read="activemq-users"
          write="activemq-users"
          admin="activemq-users" />
      </authorizationEntries>
    </authorizationMap>
  </map>
</authorizationPlugin>

What the above states is that jms service providers, such as my Camel based applications, can effectively publish and subscribe to anything. However my client belongs to the javaFlightTopic.Subscribers group and the activemq-users group and so can only consume from a specific topic and perform all required advisory services; the latter being an ActiveMQ requirement.

It is possible to express the authorisation mappings in an LDAP store as well. We'll see if the need surfaces.

Source Code

I have created an open-source project named jaasloginmodules that hosts this JAAS login module and have tested and used the classes. The CertificateLoginDirectoryRolesModule is ready for download and use.

When is a POJO no longer a POJO

2009-03-03T22:35:00.004+11:00

When is a Plain Old Java Object no longer a Plain Old Java Object?

My take on POJOs is that they are no longer POJOs when there is an import statement for something not conforming to the expression "java*.*".

I see quite a lot of frameworks and toolkits claiming benefits around POJO oriented development. However either through annotations, extension, implementation etc. they require dependencies on non JRE artifacts.

This situation is quite OK of course; I just think that it is useful to distinguish a POJO from its dependent form. For example if you're looking for Object Relational Mapping then you might want to consider the Java Persistence API instead of being directly dependent on an implementation; such as Hibernate. Maintaining this distinction allows you to consider migration strategies to other implementations if required.

Sometimes you need to be dependent on a toolkit or framework and that is of course perfectly acceptable. Being aware is the significant point.

Who is Palm trying to attract for WebOS development?

2009-02-21T09:07:00.005+11:00

I must say that my first impressions of what Palm is offering in terms of the WebOS leaves me curious.

I wrote for the Palm OS over a long period of time. I started my Palm OS development when Palm were US Robotics and programmed the original PalmPilot. The PalmPilot was an amazing achievement for the time.

There were a few things that attracted me to the original Palm OS platform, but a major one was that I could use my C++ skills to create robust and high-performance deliverables. The Metrowerks Codewarrior environment was a joy and I could use the same IDE for Mac OS (then Mac OS 8!) and Palm OS development.

I think that it is reasonable to state that the original Palm OS attracted seasoned software developers; a lot of them were also Mac developers that understood things like usability.

I am little surprised to learn that the WebOS is a Javascript world for the developer. To me Javascript is a language that evolved i.e. it does not appear to have been well thought out from the start. I am not attempting to criticise Javascript here - in fact I think that it is amazing what has been enabled via Javascript in the context of a web browser. I also understand why it has evolved the way it has. The thing is I am just surprised that Palm have chosen Javascript as their primary programming environment for WebOS. Who exactly is it that Palm are trying to attract to their platform?

I suppose what I am saying here is that while Javascript can be written in a disciplined manner, I do not often see this to be the case. I am therefore curious about the quality of applications that will be delivered to the WebOS.

I think that Apple have things almost right with the iPhone SDK and that Google have it completely right with Andriod. Where Apple need to improve is simply to provide a notification mechanism for an application. The notification handler should be able to operate in the background. The original Palm OS provided such a mechanism for handling alarm and other events. The benefit was that you could get an application to do things as a result of various device stimuli. This capability is very important and the iPhone falls short in this regard; so much so that I cannot consider porting my Palm OS based Titan Class Vision software.

In the case of Google I think that the choice of Java is correct. Java is a great way to deliver robust software and rich in its tooling. However Google have to solve the battery life implications of their environment which presently make their phone unusable in many quarters.

I digress slightly. The question remains though, who is Palm trying to attract for WebOS development?

Using Camel ProducerTemplate in a POJO

2009-02-04T08:02:00.004+11:00

I am enjoying Apache Camel immensely. My productivity in terms of delivering enterprise grade services has increased dramatically. No doubt that the quality of my deliverables has also increased. All of this with no compromise to performance...

In the case of having deployed about 3 Camel based applications (containers for services) and a couple of non-Camel ESB applications, I think that one of the major ESB benefits is the de-coupling the service implementation from the transport. This of course allows the service code to be written as a POJO and with dependency injection, DAOs and a few other patterns, the resulting service has a clean focus on the business logic.

I recently had to implement a service that initialised a biometric device. Initialisation comprised of communicating several messages over a TCP connection to each biometric device. In the first instance I used Camel's ProducerTemplate to send my messages. This all worked fine and of course meets that goal of separating out the transport delivery from the business logic. However I now had a Camel dependency in my otherwise pure POJO service. This did not satisfy my design goal of ensuring that the service remained a POJO so that I could be free with my ESB choice (not that I see myself moving from Camel for any reason!).

To avoid this Camel dependency I created and used my own ProducerTemplate interface. This is merely a proxy to Camel's and only has the methods I need from my service. It looks like this:

public interface ProducerTemplate {
 Object sendBody(String endpoint, Object body);
}

I then pass a ProducerTemplate instance and its endpoint details to the invocation of my service along with any other parameters the service needs:

void process(ProducerTemplate producerTemplate, String deviceEndpoint,
 Collection fingerprints)

Note that the ProducerTemplate is passed in on each invocation of the service. This is because the template can potentially change between service invocations.

My Camel container project includes the project housing the above POJO. The container defines and instantiates its ProducerTemplate that forwards calls on to Camel's ProducerTemplate. The result is that I now have a POJO based service that interacts with other services in a non-Camel dependent manner.

Anatomy of a service/Apache anyone?

2008-12-05T22:22:00.006+11:00

Here is a pattern that I have implemented a few times now.

I have had requirements to provide a web service, typically SOAP based, that persists something in a database (or whatever). Here's a general flow of events that I'm finding works particularly well:

1. Web browser to web service
A web application (typically Javascript executing in a web browser) sends a SOAP request. Given that my web service is written using Apache CXF it can generate the required Javascript for the browser's invocation by appending a ?js parameter to the web service. Check out the CXF doco to see how.

My CXF based web service is developed on a contract-first basis i.e. the SOAP interface is defined using Java. The reason I like that (as opposed to a WSDL-first approach), is because WSDL appears overly complex to me. It is much easier, in my most humble opinion, to create the Java interface and have CXF generate the WSDL. I reckon that CXF does a pretty good job of this too.

My web service has the responsibility of marshaling the SOAP XML into a Java object that is representative of my information model. Here's a critical point: you must have an information model to work with aka domain model. If you don't have that nutted out very well then you tend to spin wheels.

The classes of the information model are used as a normalised object for messaging and also for persistence. The web service marshals the XML into model objects, performs various integrity checks and then forwards the model objects on to a JMS queue.

I'm using Spring's JMS template classes to forward on my JMS messages; when it comes to sending messages, lots of code is taken care of using these classes (I've also programmed JMS without them so I can make a direct comparison as to what Spring offers in this respect).

In summary my web browser invokes a SOAP web service using some nice Javascript code that CXF can generate. My CXF based service has a relatively simple roll of marshaling the SOAP XML input into a domain object of my information model and sending it to a queue.

Oh, by the way, I'm using Apache Tomcat to host my CXF based web application.

2. Sending to ActiveMQ
I'm typically sending to queues dedicated to specific functions e.g. "queue to persist the biometric fingerprint of a person". These queues are persistent (a JMS default) so that messages are stored until they can be guaranteed delivery.

I like ActiveMQ. Apart from the price tag being very attractive (free), the Apache products are built to a very high standard. ActiveMQ is fast and reliable.

3. ActiveMQ replies
Once ActiveMQ replies back to my web service guaranteeing delivery, my web service replies to my browser application.

4. Browser processes the reply & Camel service consumes queue
These are concurrent activities. Of course the browser delivered the SOAP request in (1) asynchronously and registered call backs for processing any errors and successful responses. This way the user's browser remains responsive.

The design of my browser application is very much around the store-and-forward paradigm. Depending on how critical it is to feed back the results of an operation, it is generally adequate to inform the user that their requested has been posted. In the case where the request is involved in retrieving, say, customer details, receiving a response is important so I would use a request-reply enterprise integration pattern. However in many scenarios, store and forward is perfectly adequate and really suites a browser style application.

Almost as soon as ActiveMQ makes a message available to the queue (2) my Apache Camel based service is able to consume it.

I like Apache Camel. It really helps me assemble services very loosely given the strong promotion of enterprise integration patterns. You can build content based routers, aggregators, request-reply mechanisms and so forth - Camel provides the framework for you to build your services on. Camel almost forces you to think about the correct separation of concerns given its abstractions.

I typically deploy my Camel based services using jsvc - Apache's daemon toolkit (part of the Apache Commons project). jsvc is a very simple wrapper so that you can make your application run as a service on Unix and Windows based platforms. From a programming perspective, you implement a lifecycle interface (init, destroy, start, stop) and that's about it. I want to move toward using OSGi, and I have played with it. However I have not found a OSGi based container that I'm totally happy with yet (they don't appear to have totally satisfied use-cases associated with administrators - just programmers at this point).

I have many services do many things but a common task is persisting and retrieving data from a relational database. For that I program to JPA and use Hibernate as my JPA implementation. I prefer JPA over Hibernate directly so that I have the ultimate freedom of using other ORM implementations should I need to - I don't think I will be needing to at this point though as Hibernate is great.

Performance
I get great performance. The slowest parts tend to be associated with persisting data in an RDBMS (nothing to do with Hibernate etc - normal RDBMS stuff) and network latency. CXF and Camel fly along - you're mostly counting small units of milliseconds with these technologies.

Scalability
I've not had to do much in the way of scaling yet, but I know I have the correct separation of concerns to cope with most of what the world can throw at these services. In the performance testing that I have done I tend to exceed my customer's expectations. As stated above, it is network latency that appears to be the bottleneck given that the internet often sits between the browser and the web service.

Apache, Apache
It may appear that I work for Apache or something, but I don't. I do value the Apache Projects though. In summary here's what I'm presently using:

* Active/MQ
* Camel
* CXF
* Maven
* MINA
* Tomcat

Software Development and the Global Economic Downturn

2008-11-29T12:39:00.008+11:00

There appears to be plenty of software development work around at the moment.

A manager at a large company that I have been doing some work for recently commented that when a business looks to contract, the demands on IT increase.

I agree.

The reason for this is automation; something that the business should always strive for of course. The point is that they don't. Large companies would rather throw multitudes of people at a problem than optimise operational tasks... until external pressures such as world economic downturns occur. I have friends that work for some other large companies that are typically regarded as being innovative. One of the common threads is how, from an operational perspective, they always automate the mundane tasks as much as possible.

So for me at least, as a small software developer, there appears to be plenty of work around and a great deal is focused on automation.

An Image Compositor Technique for a Planet

2007-10-19T14:13:00.001+10:00

I had a need to develop a capability within Titan Class Vision to composite many images representing parts of our planet at different resolutions. In particular, these images could be quite large; perhaps 200MB each.

I decided to give memory mapped files a go using the low-level paging mechanisms available to modern operating systems. In a nutshell we’re looking at the use of mmap and munmap. Memory mapping files is very fast and highly optimised - it is by far the fasted way of reading bytes in to memory from disk.

My main image of the planet is about 200MB in size. What I do is have a Windows BMP file mapped saved in BGRAUnsignedInt8888Rev form (this is the optimal format for something to be textured on Mac OS X) and then use memory mapped file IO to page the BMP into memory. I then have an image compositor object that is able to composite many of these images e.g. I have a BMP for the entire planet, and one for just a given state of Australia (NSW). When it comes to client code making a request the compositor assembles a composited buffer of my layers as required and in the resolution required. This composited buffer is then sent to a texture using GL_STORAGE_SHARED_APPLE as an optimisation. I also keep some of these textures around for situations where I know in advance that I'm going to zoom in on something; it is then consequently very fast when it comes to rendering as no dynamic composition is required.

Oh, and I'm using the Mac OS X Accelerate Framework for high quality and high performance scaling given that Titan Class Vision targets this platform.

It all works pretty well and is fast, even on a tired old G4 Powerbook. Multiprocessors are utilised given the Accelerate Framework. One naturally has to be considerate of virtual memory usage with the compositor, but that is a resource management exercise.

Here’s the class structure that I came up with; feel free to use it in your own work but please be kind and include a reference to this page and some credits.


namespace WorldLayerCompositor {
    typedef unsigned long BGRA;
    
    inline unsigned short SwapInt16LittleToHost(
        unsigned short arg
        ) throw();

    inline unsigned long SwapInt32LittleToHost(
        unsigned long arg
        ) throw();

class Layer {
  public:
virtual ~Layer();

inline void GetCentreLatLong(
            double& outLat, double& outLong
            ) const throw();

inline double GetResolution() const throw();

inline void GetSizePx(
            unsigned& outWidthPx, 
            unsigned& outHeightPx
            ) const throw();

virtual BGRA* GetBuffer() const throw() = 0;
};

class MemoryMappedBMPFileLayer : public Layer {
  public:
MemoryMappedBMPFileLayer(
            const std::string& inFile
            ) throw();
virtual ~MemoryMappedBMPFileLayer();

virtual BGRA* GetBuffer() const throw();
};

class LayerCompositor {
  public:
LayerCompositor() throw();

typedef std::list<boost::shared_ptr<Layer> >
            LayerList;

LayerList::iterator AddLayer(
            boost::shared_ptr<Layer> inLayerP
            ) throw();

void RemoveLayer(
            const LayerList::iterator& inLayerIter
            ) throw();

bool GetNextSubBuffer(
            unsigned long** inBGRAUnsignedInt8888RevPP, 
            unsigned& outSubXPx, 
            unsigned& outSubXPy,
            unsigned& outSubWidthPx, 
            unsigned& outSubHeightPx
            ) const throw();

void GetEffectiveSizePx(
            unsigned& outWidthPx, 
            unsigned& outHeightPx
            ) const throw();

inline void SetSubRegion(
            double inLat, double inLong,
            double inResolution,
            unsigned inWidthPx, unsigned inHeightPx
            ) throw();
};
};

I intend to evolve this class much further and make it considerate of planet related things. For example if a request is presently made for a region that extends over the dateline then I move the region back either east or west. In the future I’ll be handling this so that the compositing considers the dateline.

Another thought is to be able to add layers described in vector terms using SVG and GML... I think that there are some interesting possibilities.

Using The GML’s Moving Object Status for presenting Flight Information

2007-07-16T13:56:00.001+10:00

This is something that I’ve wanted to share for sometime now - how to use the Geography Mark-up Language (GML) for tracking things that move in the real world. GML can be a lot to get one’s head around so I hope that this real-world implementation will help you. We have been using GML for sometime now with our Titan Class Vision product.

I want to show you how GML can be used to describe flight data; the sort of data you’ll see presented in the arrival and departure halls of airports. I have a hope that the world can share a view on how airport flight information should be provided to many kinds of application.

GML provides a set of abstractions that are intended to be specialised by applications for their own purpose. Consequently we have defined a schema to describe journeys. This schema is represented by the class diagram above and is available in XSD form from our website.

Journeys describe a track between point A (departure port) and point B (arrival port). They have a scheduled departure and scheduled arrival time. A specific type of journey is a flight which has no other characteristic than its type for distinguishing itself.

A Journey is a specialisation of a GML DynamicFeature. Dynamic features are things that have a relationship to geography (as per a regular Feature) but can also change over time and record when they do so. Consequently snapshots of a dynamic feature can be expressed given the availability of a validTime element. Additionally, and importantly, dynamic features can also have a “track” element. Tracks define a set of objects that describe a feature’s status over a period of time. These objects are known as GML’s MovingObjectStatus and minimally describe the time that the status applies to (validTime) and the location of the dynamic feature. A location can also be “null” i.e. unknown; this is pertinent to us and I’ll come back to it. The other elements are also very useful when tracking the position of a dynamic feature. For example, given acceleration, speed, bearing (horizontal and vertical), elevation etc. one can reasonably predict where the feature will be next. For most flight information displays in airports though, these elements are less important (Titan Class Vision at the airport is attempting to change this situation!).

We have further specialised MovingObjectStatus as a JourneyStatus with the discriminating element being the Estimated Time of Arrival (ETA). Thus for a given validTime we can report what the ETA was.

Enough words! Let’s have an example based on our journey schema:


<j:FeatureCollection ...>
<name>Journeys</name>
<boundedBy>...</boundedBy>

<featureMembers>
<j:Flight>
<name>QF32</name>
<validTime>
<TimeInstant>
    <timePosition>
        2006-05-01T10:30:00Z
    </timePosition>
</TimeInstant>
</validTime>
<track>
<j:JourneyStatus>
<validTime><TimeInstant>
        <timePosition>
            2006-05-01T10:30:00Z
        </timePosition>
</TimeInstant></validTime>
<location><Null/></location>
<j:ETA>
    2006-05-02T00:05:00Z
</j:ETA>
</j:JourneyStatus>
</track>
<j:departPort xlink:href="..."/>
<j:departTime>
   2006-05-01T11:15:00Z
</j:departTime>
<j:arrivePort xlink:href="..."/>
<j:arriveTime>
    2006-05-02T00:00:00Z
</j:arriveTime>
</j:Flight>

</featureMembers>

<validTime><TimeInstant><timePosition>
    2006-05-01T10:30:00Z
</timePosition></TimeInstant></validTime>

</j:FeatureCollection>

Please note that the above is not valid XML i.e. I’ve removed the namespace declarations and the xlink:href contents so that I could fit everything on the page and just present what is relevant to this discussion. A valid version of the above can be found on our website.

The above document states, “here are the flights as at 1030 GMT. There is just one flight named QF32 which is scheduled to arrive at midnight GMT. However the current ETA is 0005 GMT.”. In the real world there would be a number of Flight elements for a given airport and a number of JourneyStatus elements if there have been a number of ETA revisions. The location is not important for most flight information displays at airports and hence we show that as having a null value.

ETA could have been included as a element of Journey. I chose to have it as an element of JourneyStatus (aka MovingObjectStatus) so that I could have a number of ETAs reported for historical reasons. Additionally the reporting of an ETA could be further analysed by location.

Plans are deemed other features (including other Journey features) that can further describe a given Journey feature. For example we have journeys that describe the frequently used flight paths between various destinations. These paths can include various way-points and record typical elevation and speed at those points.

Our schema can also be used to describe a schedule such as a flight schedule. The validTime element of a Journey feature has the potential to be used to distinguish different schedules depending on the day of the week. For example a scheduled flight might only occur on Mondays and Fridays.

I’m hoping that it is becoming apparent that more than just the ETA of a flight can be described. Firstly the structure of our schema permits the actual positions of flights at given time instances. We have embarked on consuming air radar data using our journey schema. This will enable us to render the real-time track of aircraft with our flight information display.

Secondly anything that has a journey should be able to be described. This includes trains, ships, cars and trucks - even people swiping their security cards.

In a future blog I shall describe various forms of accessing journey data using the Web Feature Service (WFS) and also how this data can be published and subscribed using an enterprise messaging bus such as IBM’s Websphere MQ and others that use the Java Messaging Service (JMS).

Meanwhile if you find that our journey schema is useful to you then please add a comment or do get in touch.

Titan Class Vision & Quartz Composer

2007-04-26T13:43:00.006+10:00

I’ve just about finished integrating Quartz Composer with Titan Class Vision. For those of you that know nothing of Titan Class Vision, it provides a global flight information display that tracks aircraft in real-time. Our "Google Earth" type display presents new revenue generation opportunities for airports and also serves as a good airport-customer initiative. Titan Class Vision has been installed on two displays near the A/B exit of the Terminal 1 Arrivals Hall in Sydney International Airport (T1), Australia.

Before I describe my initial requirements and how I’ve integrated Quartz Composer here is a movie showing the animated integration.

I needed to consider using Titan Class Vision with Plasma Display Panels and thus minimise burn-in issues. Titan Class Vision is already quite animated zooming between various resolutions. However more movement was required to remove static text and a header and footer margin.

I could have programmed these animations into Titan Class Vision directly but I’ve always felt that something like Quartz Composer should be composing Vision’s image with other digital media and effects. By externalising most of the content (other than the planet and the 3d objects that get overlaid on to the planet e.g. the flight paths), I can now customise the display contents quite easily for individual customers - and with great effect!

Before I continue thanks to everyone on the Quartz Composer forum that have helped me over the past few weeks. In particular thanks go to “tkoelling”, Alessandro Sabatelli and Pierre-Olivier Latour.

The approach that I’ve taken is to render my 3D world into a Core Video (CV) buffer. The viewport is the size of the screen and is rendered once per frame. I then pass this CV buffer to Quartz Composer (QC). QC and CV share their OpenGL resources (context). Thus the image that I have rendered remains on the GPU side of the fence (I think that CV creates my image as a texture). QC receives the CV buffer as an image parameter and my scripts can do what they need to do. Pierre describes how to do QC/CV integration on the forum and I also posted some code there.

Titan Class Vision makes no assumption with regards to the name and location of the QC file. Instead I use AppleScript to pass in the path of the composition and this causes the instantiation of a new QCRenderer object (releasing any previously held one of course). At this time I also reset the timing for my composition if it is a different file to the last one. In summary I have some external program (which I actually call “Bootstrap”) that tells Titan Class Vision what to do - now including telling it where my composition file is. The Bootstrap is an Applescript Studio application that bundles the composition file as a resource.

Bootstrap tells Titan Class Vision to zoom through different resolutions on a periodic basis - typically every 30 seconds. It now also sets up input parameters to the QCRenderer so that parts of my composition are enabled/disabled. I typically have one composition file per client’s implementation and enable sub-patches as necessary depending on what I want to show. This approach eradicates any performance impact of instantiating a new QCRenderer object (not that there appeared to be much of an impact though). Note that I do release/alloc a new QCRenderer if my context is re-shaped due to a bug with QC taking note of its viewport only on instantiation.

For your interest, Bootstrap passes input parameters to my QCRenderer via Applescript as a URL encoded string parameter e.g.

set composition parameters to "Fade_In=1&Fade_Out=0&World_Actual_Size=1&World_Zoomed=0"

That’s about it from a high level perspective other than to say that I can easily get 60fps on my relatively slow development machine (dual G4 1Ghz). I haven’t released my changes to my Quad Xeon yet but will do so in the next couple of months and all of this should fly (pun intended)!

Please note that when rendering to a CVOpenGLBuffer it must be treated as something that is immutable i.e. after you have written to it and passed it to the QCRenderer do not try writing to it again. This is in fact exactly what I was doing on Tiger and all was well. However with Leopard I had a problem given that the QC imaging pipeline had apparently changed.

The resolution is to use a CVOpenGLBufferPool and then use CVOpenGLBufferPoolCreateOpenGLBuffer each time you want to render a new frame. After the frame has been passed to the QCRenderer you then release the CVOpenGLBuffer. Problem solved.