tag:blogger.com,1999:blog-14133043120419766592024-03-15T05:49:20.219+11:00Christopher Hunt on Software DevelopmentThis is my blog where I like to focus on things of a software development nature.Christopherhttp://www.blogger.com/profile/06860404627954267472noreply@blogger.comBlogger64125tag:blogger.com,1999:blog-1413304312041976659.post-72160108455549663652023-06-12T13:49:00.004+10:002023-06-12T13:54:43.551+10:00Introducing "standard streamed commands and events", or "stdce"<p>This blog introduces a stream-based command/event programming model that can be applied to many programming languages. Having an event-driven approach to programming like this helps achieve resiliency and responsiveness. Resiliency should speak for itself, and responsiveness often leads to an improved user experience.</p><p>Since writing up on <a href="http://christopherhunt-software.blogspot.com/2021/02/event-driven-finite-state-machines.html">Event-drive Finite State Machines</a> a couple of years back, colleagues and I have worked on <a href="https://github.com/titanclass/edfsm">edfsm</a> and the <a href="https://github.com/titanclass/flip-flop-protocol">flip-flop protocol</a>. "edfsm" is a Rust implementation of that previous write-up, and the flip-flop protocol is a half-duplex protocol for conveying commands and events. Commands and events are the common denominators to all that I've just referred to and form the foundation of what I'd like to introduce here today: the formality of a programming model that has surfaced for me, and one that can be used to shape many types of program, including those solved presently by "<a href="https://en.wikipedia.org/wiki/Function_as_a_service">function-as-a-service</a>" (FaaS). That's the last mention of FaaS here though; I don't want this idea to be constrained to this one, albeit popular approach to programming!</p><p>The approach I wish to formalise uses <a href="https://en.wikipedia.org/wiki/Standard_streams">standard streams</a> (stdio) and is essentially this:</p><p></p><ul style="text-align: left;"><li>stdin is used to receive a continuous stream of commands</li><li>stdout is used to emit a continuous stream of events where events can be of type "logged" or "ephemeral"</li></ul><div>...and that's it.</div><div><br /></div><div>The approach enables long-running event-driven programs to be written in virtually any programming language that supports stdio, and most programming languages do. Furthermore, the approach conveys the notion that events can be stored in an event log (logged), or that they may come and go where their loss is insignificant (ephemeral).</div><div><br /></div><div>Pipelines can also be achieved given stdio. For example, a program hosted in an <a href="https://opencontainers.org/">OCI</a> container might be communicated with via <a href="https://en.wikipedia.org/wiki/Unix_domain_socket">Unix Domain Sockets</a> producing and consuming with the program's stdio.</div><div><br /></div><div>Using stdio to receive a stream of commands and emit a stream of events may not even be so novel. I'd be interested to learn of similar efforts and make comparisons. That said, this particular blog isn't going to fully formalise the approach either. That's better served with examples. But, I would like to capture a name for the approach and introduce it at a high level. To that end, I'm going to name this approach "standard streamed commands and events", or "stdce" for short.</div><div><br /></div><div>Based on the work I've done in this space already, here are some commands and events that may be required. There will probably be more!</div><h2 style="text-align: left;">Commands</h2><div>A command is delivered on stdin as JSON using the following form:</div><div><br /></div><div><div><span style="font-family: courier;">{</span></div><div><span style="font-family: courier;"> "command": "<some-command>"</span></div><div><span style="font-family: courier;"> "args": {</span></div><div><span style="font-family: courier;"> "<some-arg>": <some-arg-value></span></div><div><span style="font-family: courier;"> }</span></div><div><span style="font-family: courier;">}</span></div></div><div><br /></div><div>The following commands are envisaged:</div><div><ul style="text-align: left;"><li>Configure</li><li>HandleRecoveryEvent</li><li>WaitForEvents</li><li>HandleEvent</li></ul><div style="text-align: left;">The Configure command is the first command received and provides any secrets and other configurations that the program may require. I've previously used secrets to decrypt command payloads and encrypt event payloads.</div><div style="text-align: left;"><br /></div><div style="text-align: left;">HandleRecoveryEvent is sent for each event sourced from an associated log i.e. previously "logged" events. A program may reconstitute its state from these recovery events. A WaitForEvents command will be sent when all recovery events have been sent.</div><div style="text-align: left;"><br /></div><div style="text-align: left;">HandleEvent is sent for each event that occurs post-recovery. Events may be sourced from many topics including those produced by other processes.</div><h2 style="text-align: left;">Events</h2></div><div>An event is a JSON output to stdout using the following form:</div><div><span style="font-family: courier;"><br /></span></div><div><div><span style="font-family: courier;">{</span></div><div><span style="font-family: courier;"> "event": "<some-event>"</span></div><div><span style="font-family: courier;"> "type": "<event-type>"</span></div><div><span style="font-family: courier;"> "args": {</span></div><div><span style="font-family: courier;"> "<some-arg>": <some-arg-value></span></div><div><span style="font-family: courier;"> }</span></div><div><span style="font-family: courier;">}</span></div></div><div><br /></div><div>Any event that the program wishes to emit can be conveyed using an event type of "Logged" or "Ephemeral".</div><h2 style="text-align: left;">Configuration</h2><div>Not mentioned so far, but of course, a program will need to provide a configuration in terms of "topics" to subscribe to namespaces and other things. This information can be communicated outside of stdce and is left as an implementation concern.</div><h2 style="text-align: left;">Next steps</h2><div>This blog is a starting point, and somewhere for me to just get the idea down and name it. I'd like to get some feedback and then expand by fleshing out a project on GitHub. Hopefully, more to come soon then...</div><p></p>Christopherhttp://www.blogger.com/profile/06860404627954267472noreply@blogger.com0tag:blogger.com,1999:blog-1413304312041976659.post-67856827715170565032021-02-07T14:38:00.003+11:002021-02-17T17:45:18.466+11:00Event-driven Finite State Machines<p>I often use Event-driven Finite State Machines to capture stateful behaviour. Doing so in combination with a compiler with a strong type system helps me to think about all possible combinations of the effects of commands on a given state, and what the events produced (if any) are. In particular, I find many programs required for embedded applications can often be safely expressed by using Event-driven Finite State Machines, leading to very high-quality outcomes. They're also great for managing those stateful scenarios where network communication takes place.</p><p>My intention here is to capture a pattern that I've found myself using a great deal. I didn't come up with this pattern, and in fact for me, this pattern evolved out of using Akka persistence for event sourcing (1). These are also known as Event-driven finite state machines, which are described as the "transition from one state to another is triggered by an event or a message" (2).</p><p>Here are the two function signatures that I use:</p><p><span style="font-family: courier;">fn for_input(s: State, c: Command) -> Vec<Event><br /></span></p><p><span style="font-family: courier;">fn transition(s: State, e: Event) -> State</span></p><p>The first function yields zero or more events to be caused, given a state and a command. The second function applies an event given a state to yield a new state.</p><p>Commands are an enumeration of actions that can be performed on the state machine and defined exclusively within the confines of it. Events can come from many places outside of the state machine and cause commands to manifest themselves for the for_inputs function. For example, the state machine might not be aware of a network, yet network events such as the receipt of data can manifest in one or more commands to submit to the for_inputs function.</p><p>The for_input function yields zero or more events having carried out the command, which may be side-effecting.</p><p>For each event returned from the for_input function, the transition function is then called, yielding a new state. The transition function avoids performing side-effects and is therefore pure. We should be able to replay events to produce a state without causing side-effects. More on that later.</p><p>These functions are able to be combined to perform commands and transition state e.g.:</p><p><span style="font-family: courier;">fn run(s: State, c: Command) -> (Vec<Event>, State)</span></p><p>However, I find it useful to keep both of the functions distinct even when composed, as it helps think through:</p><p></p><ul style="text-align: left;"><li>what commands upon state produce what events; and</li><li>what events cause a transition from one state to another.</li></ul><p></p><p>These considerations map nicely to FSM diagrams e.g.</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjSnnLJ6a2NEI8PrgOyk91X5EEm_ibMGn_qXQnKtRV-ntKW1f1NldeeLU-30eWuw1fNzG38JxWeGjF7ljugHi3_swO_2THGQqRKMLKA4eSrr0YpsRv6whJJ890VBBeS6Fy0NftTGiV_wSJN/s796/Screen+Shot+2021-02-07+at+2.04.49+pm.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="796" data-original-width="178" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjSnnLJ6a2NEI8PrgOyk91X5EEm_ibMGn_qXQnKtRV-ntKW1f1NldeeLU-30eWuw1fNzG38JxWeGjF7ljugHi3_swO_2THGQqRKMLKA4eSrr0YpsRv6whJJ890VBBeS6Fy0NftTGiV_wSJN/s320/Screen+Shot+2021-02-07+at+2.04.49+pm.png" /></a></div><br /><div class="separator" style="clear: both; text-align: center;"><br /></div>Here, the State enumeration is Initial and Idle, the Commands are Start and Stop and the events are Started and Stopped. Given the Initial state, if a Start command is received then a Started event should be produced, causing a transition to the Idle state.<br /><p>Writing the code for these two separate concerns also maps quite nicely. Keeping the diagram up to date with the code, and treating it as the source-of-truth, pays dividends in communicating design and driving modifications to the code.</p><p>Another aspect of having the two functions is that the second function permits the restoration of the state from replaying a series of events. This is known as event-sourcing (3). In a nutshell, event sourcing permits us to reconstruct the state by replaying the events historically. We can therefore avoid replaying commands and their effects and call just the transition function. By leveraging event sourcing, we can also persist our events having called the for_input function, and subsequently restore our state on the state machine even when the host program is restarted.</p><p>States, Commands, and Events are often expressed in a language using enumerations. By leveraging pattern matching in languages such as Rust, Scala, and Kotlin, the compiler can help us ensure that we are thinking of all combinations of applying commands to state and events to the state. Resist the temptation to "catch-all" commands or events by using matching expressions (match arms) as it is inevitable that they'll be a state combination that you've not considered, and we want the compiler to help us exhaust them all. That said, there some commands that apply to all states. In our example, the Stop command may be universally applied no matter what the state. There are always exceptions.<br /></p><p>The state objects will contain, well, state! This internal state can be shared between state objects leveraging the fact that there will only ever be one state at a time. Reference counting this internal state is common where value-copying may be expensive. In Rust, I've successfully used <span style="font-family: courier;">Rc<RefCell<T>></span> to share mutable internal states between each State enumeration.</p><p>Happy stateful programming!</p><p>(1) <a href="https://doc.akka.io/docs/akka/current/typed/persistence.html">https://doc.akka.io/docs/akka/current/typed/persistence.html</a><br />(2) <a href="https://en.wikipedia.org/wiki/Event-driven_finite-state_machine">https://en.wikipedia.org/wiki/Event-driven_finite-state_machine</a><br />(3) <a href="https://martinfowler.com/eaaDev/EventSourcing.html">https://martinfowler.com/eaaDev/EventSourcing.html</a></p><p><br /></p>Christopherhttp://www.blogger.com/profile/06860404627954267472noreply@blogger.com0tag:blogger.com,1999:blog-1413304312041976659.post-17148499568955063832020-01-28T12:14:00.004+11:002020-01-28T12:14:43.619+11:00Conflict-free replicated data as an application concernA while back, I wrote about <a href="https://christopherhunt-software.blogspot.com/2019/07/converging-data-in-iot.html">converging observations in IoT </a>and described how we de-duplicate sensor data. This post is a follow-on in that it describes the same domain in terms of how we distribute the data itself. My inspiration for this particular post is from Colin Breck's <a href="https://twitter.com/breckcs/status/1203736751681896449">excellent post </a>on "Shared-Nothing Architectures for Server Replication and Synchronization". In particular, Colin describes the development of a distributable event log that he was involved with prior to the days of Kafka. We also have a distributed event log so I thought it'd be great to share our approach given that distributing data is hard and we have adopted a relatively simple solution, no doubt used by a few others.<br />
<br />
In a nutshell, we move the problem of distributing data to the domain of the application. The closer this problem is to the consumer of the data then the easier it seems to handle (note: I'm not saying it is easy, just easier!). My previous article on converging observations is pretty much of the same vein; de-duplicating data at the point of consumption is much less error-prone than de-duplicating at the source given that there can be many sources that would otherwise need to know of each other.<br />
<br />
A real-world example is needed.<br />
<br />
We have a topic that events are appended to on a durable queue (our event log). These events describe IoT devices as distinct from recording observations. We call these events "end device events". Here's a complete list of them:<br />
<br />
<ul>
<li>BatteryLevelUpdated</li>
<li>NameUpdated</li>
<li>NoncesUpdated</li>
<li>NwkAddrUpdated</li>
<li>NwkAddrRemoved</li>
<li>PositionUpdated</li>
<li>SecretsUpdated</li>
</ul>
<div>
(if you're interested, we have a <a href="https://streambed.io/operations/reference/control-center/api/index.html">public API </a>to our system describing these events and more)</div>
<div>
<br /></div>
<div>
Our system has a cloud-based component and an edge-based one. We provision end devices in the cloud and record their observations at the edge (the edge is a farm in the case of our agriculture business).</div>
<div>
<br /></div>
<div>
NwkAddrUpdated, NwkAddrRemoved, and SecretsUpdated can only occur in the cloud. This is a design decision and reflects the use-case of provisioning end devices. When the edge connects to the cloud, it receives these events which are sufficient for an end-device to start sending its observations at the edge. The edge-based end-user may subsequently associate a position and a name with the device. As we receive data from an end-device, battery levels and nonce values (for end device authentication) will also appear as events. These edge-based events are pushed to the cloud.</div>
<div>
<br /></div>
<div>
This single source of writing, at a fine level of granularity, results in a complete picture of end devices both in the cloud and at the edge - with no conflict resolution required!</div>
<div>
<br /></div>
<div>
Now, the doubters among you may be saying, "but what if I need to provide the position from both the cloud and the edge"... or something similar... Well, then you might have a PositionReset event for representing the position provided to the cloud i.e. create a new event type!</div>
<div>
<br /></div>
<div>
Thus... <i><b>event types dictate where they are written</b></i>, hence my earlier assertion that distribution becomes an application concern.</div>
<div>
<br /></div>
<div>
We can also replicate events in the cloud for the purposes of backup and sharding. A nice feature of durable queues is that consumers can remember the offset of the last event that was consumed. Each consumer can, therefore, operate at their own pace, which is particularly useful with replicating between the cloud and the edge as the networks connecting them are typically unreliable. Replicating within the cloud is typically very fast and barely noticeable as the networks there are much faster and more reliable.</div>
<div>
<br /></div>
<div>
When writing events in the cloud we can use a partition key to determine which node is responsible for receiving the event. Again, there's only one node responsible for writing.</div>
<div>
<br /></div>
<div>
The down-side of our approach is that if a node becomes lost, at the edge or in the cloud, then the associated events will be lost. This is acceptable for our use-case though as the mean-time between failures is relatively low. </div>
<div>
<br /></div>
<div>
The approach we've taken works for our use-cases in the world of IoT where best-effort is quite acceptable as the network itself tends to be quite unreliable. There are trade-offs but, I believe, the above approach can also work for many use-cases outside of IoT.</div>
<div>
<br /></div>
<div>
Thanks for reading.</div>
Christopherhttp://www.blogger.com/profile/06860404627954267472noreply@blogger.com0tag:blogger.com,1999:blog-1413304312041976659.post-14905754777885481842019-07-19T09:19:00.000+10:002019-07-19T09:21:38.767+10:00Converging observations in IoTThe problem is this. You have a sensor on a farm that transmits its observations. The observations can be received by one or more towers. Each tower forwards the observations it receives to a server that then stores them. You now have a situation where a single observation has been recorded twice. What to do?<br />
<br />
You might choose to de-duplicate at the server. You know that there can only be one observation sent per hour, so if there are two within the hour that have the same values then pick just one. Easy.<br />
<br />
However, what if you then have multiple servers, perhaps for redundancy? What then if one tower connects to one server, and another tower connects to another server? It could even be that a third party server sends us the same observations given a roaming style of arrangement. Same observations, two towers, two servers.<br />
<br />
This is our reality.<br />
<br />
You might employ clustering for your server and attempt to eliminate the duplicate state. Conflict-free Replicated Data Types (CRDTs) are great for this. This is a reasonable solution and having used CRDTs lovingly, we could just stop there.<br />
<br />
We’ve chosen another strategy though, similar in spirit to CRDTs. However, we think it may be a bit simpler and it also allows for the other server to not be under our control as per the roaming scenario.<br />
<br />
CRDTs have this wonderful property of always knowing how to merge i.e. there is never any conflict. Our view is that we permit conflict and eliminate it at the point of being consumed. We allow observations to be recorded indiscriminately as we don’t know what other systems will also be recording them.<br />
<br />
The “point of being consumed” is most often when we render the data at the presentation layer. We provide the presentation layer with multiple sources of observations. We let the presentation layer de-duplicate. This is powerful as the presentation layer understands the context in which it is consumed. It is easy for it to reason that an observation that has the same value within the same hour is a duplicate and so it can be dropped. A presentation layer is a state machine.<br />
<br />
Other consumers of observations e.g. a service that actuates given sensor inputs, is also a state machine and is also in a great position to de-duplicate. Its time window for duplicate detection could also be different from that of the presentation layer given that actuation may occur over a shorter or longer period of time.<br />
<br />
Oh, and if you’re at all worried about the number of duplicates being sent to the presentation layer then: a) generally don’t worry (measure the effect); or b) de-duplicate further upstream.<br />
<br />
And that’s it really.Christopherhttp://www.blogger.com/profile/06860404627954267472noreply@blogger.com0tag:blogger.com,1999:blog-1413304312041976659.post-37852260885105979392017-11-30T13:33:00.000+11:002017-11-30T13:33:05.799+11:00Landlord can reduce the cost of running microservicesWhen running <a href="https://github.com/spring-projects/spring-petclinic">Spring's PetClinic</a> <a href="https://projects.spring.io/spring-boot/">Spring Boot</a> reference application, my experimental <a href="https://github.com/huntc/landlord">Landlord project</a> appears to save 5 times the memory for the second instance of it. My terse observations show that PetClinic requires about 250MiB of resident memory when running as a standalone Java 8 application. When running two instances of it hosted by Landlord, approximately 342.2 MiB of resident memory is consumed in total i.e. Landlord + PetClinic + PetClinic.<br />
<br />
These observations illustrate that sharing JVM machinery has positive impacts on hosting services. While running multiple PetClinic applications on the same machine isn't a particularly realistic scenario, running multiple Spring Boot microservices on the same machine is. Landlord could, therefore, provide a significant cost reduction in the hosting of any JVM based microservices.<br />
<h2>
Introduction</h2>
I've been experimenting with a project I created named "Landlord". The idea of the project is to promote a multi-tenant JVM scenario with the goal of saving memory. Our standard usage of the JVM isn't particularly kind regarding memory usage with a simple Hello World application consuming about 35MiB of memory out of the box. This figure is about 10 times what you get with a native target. For example, the same program built via <a href="https://scala-native.readthedocs.io/en/latest/">Scala Native</a> will consume about 4.5MiB of memory. Note that we're talking about resident memory - not the JVM heap (which will be much less than that).<br />
<br />
I thought that it'd be fun to run the standard Spring Boot PetClinic application within Landlord in order to get a feel for Landlord's cost savings.<br />
<h2>
Steps</h2>
<ol>
<li>Clone PetClinic and then perform a <span style="font-family: "courier new" , "courier" , monospace;">./mvnw package</span> in order to get a standalone jar.</li>
<li>Let's determine the smallest amount of RAM that PetClinic can get away with comfortably. For this, I kept running the following java command until I stopped seeing Out Of Memory (OOM) exceptions from Java.<br /><br /><span style="font-family: "courier new" , "courier" , monospace;">java \<br /> -XX:+UseSerialGC \<br /> -Xss512k \<br /> -XX:-TieredCompilation \<br /> -XX:CICompilerCount=1 \<br /> -XX:MaxRAM=80m \<br /> -cp ${pwd}/target/spring-petclinic-1.5.1.jar \<br /> org.springframework.boot.loader.JarLauncher</span><br /><span style="font-family: "times" , "times new roman" , serif;">Note that the serial GC and other options are the results of my previous investigations in order to keep JVM resident memory usage down. There are pros/cons, but the above configuration is useful when deploying to small devices such as network gateways. That said, if you can get your application running well with the above options, you're likely to run even better at a larger scale, and potentially save money given a lesser number of machines to host your required load. One more option I'd normally use is </span>-Xss256k, but I observed a stack overflow so it seems that Spring likes lots of stack.</li>
<li>When I got to that point, I then profiled the process in order to observe the JVM heap used vs allocated vs the limit. With the above configuration, PetClinic appeared to function and I didn't observe OOM, but observing the JVM heap revealed:<br /><br /><span style="font-family: "courier new" , "courier" , monospace;">Used: 37MB Alloc.: 38MB Limit: 38MB</span><br /><br />That feels a bit too close to comfort and seemed to be causing the GC to kick in each time that I refreshed the page.<br /><br />So, I ended up specifying <span style="font-family: "courier new" , "courier" , monospace;">-XX:MaxRAM=100m</span>. This then yielded:<br /><br /><span style="font-family: "courier new" , "courier" , monospace;">Used: 46MB Alloc.: 48MB Limit: 48MB</span></li>
<li>Now, even though I've specified max ram as 100MiB, this turns only to be a starting point for the JVM on how it should size its memory regions. On OS X, if I use the Activity Monitor's inspection for a process (double-click on a process in its memory tab) then the following is reported: <span style="font-family: "courier new" , "courier" , monospace;">Real Memory: 259.3 MB</span> (that's its Resident Set Size - RSS). So, even though we stated that we didn't want more than 100MiB, this is not a limit and the JVM can take more. Apparently, this is a JVM implementation consideration. I assume (hope) that the <span style="font-family: "courier new" , "courier" , monospace;">-XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap</span> are much stronger than <span style="font-family: "courier new" , "courier" , monospace;">-XX:MaxRAM</span>, or perhaps this is just some OS X JVM implementation thing... I do wish that the JVM was more predictable in this regard.</li>
<li>Time for Landlord. Let's start it up with the same options:<br /><br /><span style="font-family: "courier new" , "courier" , monospace;">JAVA_OPTS="-XX:+UseSerialGC -Xss512k -XX:-TieredCompilation -XX:CICompilerCount=1 -XX:MaxRAM=100m" \<br />daemon/target/universal/stage/bin/landlordd \<br /> --process-dir-path=/tmp/a</span></li>
<li>Let's observe its memory via the Activity Monitor: <span style="font-family: "courier new" , "courier" , monospace;">Real Memory: 133.8 MB</span>.</li>
<li>Now to load PetClinic. For this, we need a script to load it into Landlord:<br /><br />
<style type="text/css"> p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica} </style>
<div class="p1">
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">#!/bin/sh</span></div>
<div class="p1">
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">( \</span></div>
<div class="p1">
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"><span class="Apple-converted-space"> </span>printf "l" && \</span></div>
<div class="p1">
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"><span class="Apple-converted-space"> </span>echo "-cp spring-petclinic-1.5.1.jar org.springframework.boot.loader.JarLauncher" && \</span></div>
<div class="p1">
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"><span class="Apple-converted-space"> </span>tar -c -C $(pwd)/target spring-petclinic-1.5.1.jar && \</span></div>
<div class="p1">
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"><span class="Apple-converted-space"> </span>cat <&0 \</span></div>
<div class="p1">
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"><span class="Apple-converted-space"> </span>) | nc -U /var/run/landlord/landlordd.sock</span></div>
</li>
<li><div class="p1">
<span style="font-family: "times" , "times new roman" , serif; font-size: small;">Upon executing the above script: </span><span style="font-family: "courier new" , "courier" , monospace; font-size: small;">Real Memory: 278.3 MB</span><span style="font-family: "times" , "times new roman" , serif; font-size: small;">. That's just 19MiB more than when it was run as a standalone application. Connecting to Landlord via YourKit shows the heap as:<br /><br /><span style="font-family: "courier new" , "courier" , monospace;">Used: 47MB Alloc.: 48MB Limit: 48MB</span><br style="font-family: Times;" /><br />...which is quite similar to before. There doesn't appear to be any GC thrashing either. This shouldn't be any great surprise. Its thread and heap usage is quite minimal.<br /><br />Now, using Landlord for hosting just one application is not really going to give you any great benefit. Landlord's benefit's kick in when multiple applications are run. Let's run another PetClinic within Landlord.</span></div>
</li>
<li><div class="p1">
<span style="font-family: "times" , "times new roman" , serif; font-size: small;">First, so that PetClinic's ports don't clash, declare a random port to bind to within src/main/resources/application.properties:</span></div>
<div class="p1">
<span style="font-family: "courier new" , "courier" , monospace; font-size: small;">server.port=0</span></div>
</li>
<li>Package the app via <span style="font-family: "courier new" , "courier" , monospace;">./mvnw package</span> and then invoke the Landlord script for PetClinic as before.<br /><br />Unfortunately, this doesn't work... the embedded Tomcat of Spring Boot throws an exception:<br /><br /><div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Caused by: java.lang.Error: factory already defined</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><span style="white-space: pre;"> </span>at java.net.URL.setURLStreamHandlerFactory(URL.java:1112) ~[na:1.8.0_131]</span><br />
<br />
While we're at it, there are a couple of places where Spring declares shutdown hooks. Landlord warns of this with the following output:<br />
<br />
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Warning: Shutdown hooks are not applicable within landlord as many applications reside in the same JVM. </span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Declare a `public static void trap(int signal)` for trapping signals and catch `SecurityException` </span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">around your shutdown hook code.</span></div>
<div>
<br />
Clearly, there are some changes in order for an application to run within a multi-tenant environment. The PetClinic/Spring Boot environment is built to assume that it is running within its own process. Going forward, I believe it would be easy for the Spring Boot project to cater for these multi-tenancy concerns. For now, we change the PetClinic application to use Jetty instead of Tomcat. To do this, we follow <a href="https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#howto-use-jetty-instead-of-tomcat">the recipe of Spring Boot's documentation</a>.</div>
</div>
</li>
<li><div>
Once the Jetty version is running, I observed the native Java process with a: <span style="font-family: "courier new" , "courier" , monospace;">Real Memory: 245.9 MB</span>. Under Landlord, the same package + Landlord: <span style="font-family: "courier new" , "courier" , monospace;">Real Memory: 290.9 MB</span>. A bit more of a difference to the 278.3 MiB for the Tomcat based package + Landlord, but who knows what the JVM is doing... perhaps we can assume this as being some JVM anomaly.<br />
<br />
Now, if we try to run another PetClinic within Landlord then we get an OOM memory error. Clearly, we need more JVM not having very much at hand before. Let's re-run Landlord with <span style="font-family: "courier new" , "courier" , monospace;">-XX:MaxRAM=120m</span> (20MiB more overall).<br />
<br />
We now get a problem given a clash of JMX endpoints, so we turn them off (src/main/resources/application.properties: <span style="font-family: "courier new" , "courier" , monospace;">spring.jmx.enabled=false</span>) and try again.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">Real Memory: 342.2 MB</span><br />
<br />
That's just 51.3MiB additional RSS to run what would be 245.9 MiB to run an additional PetClinic outside of Landlord. Landlord, at least in this simple observation, is reducing the memory cost by about a factor of 5.</div>
</li>
</ol>
This has been a simple test and I welcome feedback on improving it.Christopherhttp://www.blogger.com/profile/06860404627954267472noreply@blogger.com0tag:blogger.com,1999:blog-1413304312041976659.post-72292393301566610832016-10-16T18:41:00.000+11:002016-10-16T18:48:58.890+11:00When closed source should become openI've been thinking for some time about when closed source should become open, particularly in the context of when your core business is about producing software. If your core business is to provide a service such as movies, as in the case of Netflix, then the dynamics are different. Because the core business is to produce movies then simply go OSS and reap the benefits from having done so (as Netflix indeed have).<br />
<br />
Before I start I should state that my views here don't describe the only reason why to go with open software; there can be other reasons of course. Indeed there are many valid reasons to start with open as well. This post just investigates the closed to open transition, and when to make it.<br />
<br />
When your business is about producing software, you're producing software assets that contain costly intellectual property. I'm a massive fan of open software and I've made many contributions in that space. However a software business also needs to make money of course.<br />
<br />
I assert that there is a very limited window of opportunity for a software business to retain a software asset as closed; and that window is governed by the open competition that it faces. The job of the software business then, is to stay ahead of the open curve, yet yield to open software when it starts to become a threat. This happened with Java when it was threatened by <a href="https://en.wikipedia.org/wiki/Apache_Harmony">Apache Harmony</a>. I believe that Harmony subsequently died precisely because of Sun OSS-ing Java.<br />
<br />
I should state right now that my thoughts have been influenced by <a href="https://www.youtube.com/watch?v=gY0Xus4mXzQ">Danese Cooper who gave a great talk on this very subject during Scala Days 2015</a>. Denese discussed why open languages win, and I think her talk has a wider application.<br />
<br />
When discussing the subject of open vs closed software with colleagues at <a href="http://www.lightbend.com/">Lightbend</a> over the past year or two, I've described closed software as resting on a <a href="https://en.wikipedia.org/wiki/List_of_tectonic_plates">tectonic plates</a>. As these plates move around then the closed software at the edge falls off into the abyss of open software! I think that the analogy is mostly useful though in order to illustrate that the world changes. Because of this if you must regularly re-evaluate the competition that is open. If you have closed software solving a particularly useful/important problem then you can be fairly certain that open software will rise around it (again thinking of what Denese said here).<br />
<br />
Open your commercial software and neutralise its open competition, also reaping the benefits of having gone open. Focus on adding higher level value building out from your core. Stay ahead of the game.<br />
<br />
You certainly can't sit still.Christopherhttp://www.blogger.com/profile/06860404627954267472noreply@blogger.com0tag:blogger.com,1999:blog-1413304312041976659.post-58683545426891744692016-07-14T15:32:00.003+10:002016-07-14T15:43:04.301+10:00Microservices: from development to production<div class="separator" style="clear: both; text-align: center;">
<iframe allowfullscreen="" class="YOUTUBE-iframe-video" data-thumbnail-src="https://i.ytimg.com/vi/5qbX7UwuMYM/0.jpg" frameborder="0" height="266" src="https://www.youtube.com/embed/5qbX7UwuMYM?feature=player_embedded" width="320"></iframe></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div dir="ltr" style="margin-bottom: 16pt; margin-top: 0pt;">
<span style="font-family: "arial";"><span style="font-size: 14.6667px; line-height: 20.24px; white-space: pre-wrap;">Let’s face it, microservices sound great, but they’re sure hard to set up and get going. There are service gateways to consider, setting up service discovery, consolidated logging, rolling updates, resiliency concerns… the list is almost endless. Distributed systems benefit the business, not so much the developer.</span></span><br />
<span style="font-family: "arial";"><span style="font-size: 14.6667px; line-height: 20.24px; white-space: pre-wrap;"><br /></span></span><span style="font-family: "arial";"><span style="font-size: 14.6667px; line-height: 20.24px; white-space: pre-wrap;">Until now.</span></span><span style="font-family: "arial";"><span style="font-size: 14.6667px; line-height: 20.24px; white-space: pre-wrap;"><br /></span></span><br />
<span style="font-family: "arial";"><span style="font-size: 14.6667px; line-height: 20.24px; white-space: pre-wrap;">Whatever you think of <a href="http://www.scala-sbt.org/">sbt</a>, the primary build tool of <a href="http://www.lagomframework.com/documentation/1.0.x/java/Home.html">Lagom</a>, it is a powerful beast. As such we’ve made it do the heavy lifting of packaging, loading and running your entire Lagom system, including Cassandra, with just one simple command:</span></span><span style="font-family: "arial";"><span style="font-size: 14.6667px; line-height: 20.24px; white-space: pre-wrap;"><br /></span></span><br />
<br />
<div style="text-align: center;">
<span style="font-family: arial; font-size: x-large; line-height: 20.24px; white-space: pre-wrap;">sbt> install</span></div>
<span style="font-family: "arial";"><span style="font-size: 14.6667px; line-height: 20.24px; white-space: pre-wrap;"><br /></span></span><br />
<span style="font-family: "arial";"><span style="font-size: 14.6667px; line-height: 20.24px; white-space: pre-wrap;">This “install” command will introspect your project and its sub-projects, generate configuration, package everything up, load it into a local <a href="http://conductr.lightbend.com/docs/1.1.x/Home">ConductR</a> cluster and then run it all. <b>Just. One. Command.</b> Try doing that with your >insert favourite build tool here<<insert build="" favourite="" here="" tool="">!</insert></span></span><br />
<span style="font-family: "arial";"><span style="font-size: 14.6667px; line-height: 20.24px; white-space: pre-wrap;"><insert build="" favourite="" here="" tool=""><br /></insert></span></span><span style="font-family: "arial";"><span style="font-size: 14.6667px; line-height: 20.24px; white-space: pre-wrap;">Lower level commands also remain available so that you can package, load and run individual services on a local ConductR cluster in support of getting everything right before pushing to production.</span></span><br />
<span style="font-family: "arial";"><span style="font-size: 14.6667px; line-height: 20.24px; white-space: pre-wrap;"><br /></span></span><span style="font-family: "arial";"><span style="font-size: 14.6667px; line-height: 20.24px; white-space: pre-wrap;">Lagom is aimed at making the developer productive when developing microservices. The ConductR integration now carries that same goal through to production.</span></span><br />
<span style="font-family: "arial";"><span style="font-size: 14.6667px; line-height: 20.24px; white-space: pre-wrap;"><br /></span></span><span style="font-family: "arial";"><span style="font-size: 14.6667px; line-height: 20.24px; white-space: pre-wrap;">Please watch the 8 minute video for a comprehensive demonstration, and be sure to visit <a href="http://www.lagomframework.com/documentation/1.0.x/java/ConductR.html">the “Lagom for production” documentation</a> in order to keep up to date with your production options. While we aim for Lagom to run with your favourite orchestration tool, we think you’ll find the build integration for ConductR hard to beat. Finally, you can focus on your business problem, and not the infrastructure to support it in production.</span></span><br />
<span style="font-family: "arial";"><span style="font-size: 14.6667px; line-height: 20.24px; white-space: pre-wrap;"><br /></span></span><span style="font-family: "arial";"><span style="font-size: 14.6667px; line-height: 20.24px; white-space: pre-wrap;"> Enjoy!</span></span></div>
Christopherhttp://www.blogger.com/profile/06860404627954267472noreply@blogger.com0tag:blogger.com,1999:blog-1413304312041976659.post-79374490271449053862016-07-05T13:54:00.001+10:002016-07-05T13:54:51.480+10:00Developers need to care about resiliency<i>Disclaimer: I'm the technical lead for <a href="http://www.lightbend.com/products/conductr">Lightbend ConductR</a> - a tool that focuses on managing distributed systems with a key goal of resiliency.</i><br />
<br />
I've been doing a reasonable amount of travelling over the past few years. Overall I enjoy it; I don't think that I'm away that much that it has become painful - may be 3-6 international flights per year.<br />
<br />
One of the things you hear about when travelling is missing an international connection. I've been fortunate in that this has happened just once; a couple of weeks ago in fact.<br />
<br />
The airline was <a href="http://www.britishairways.com/travel/home/public/en_au">British Airways</a> (BA), and they did a really good job of trying to make up time given that flights out of Heathrow were causing delays across Europe. Thus my flight from Berlin TXL to London LHR was about two hours late. I missed my Sydney SYD flight from LHR. The BA staff did a great job of putting me up in a hotel overnight and getting me on to the next available flight. Honestly, from a staff perspective, BA were fantastic in fact.<br />
<br />
What was frustrating though was that it took about two hours to arrange the accomodation and flight booking. This was also considering that I didn't have to queue for long and that a staff member attended to me in a reasonable time frame. The problem was the BA computer system.<br />
<br />
Apparently BA have some new system. There were IT staff walking around helping the front-of-house staff get into the system and deal with its incapacity to handle any load. The IT staff were frustrated. The front-of-house staff were frustrated. I was frustrated (although not as frustrated as a Business Class passenger in front of me who felt that his ticket meant that BA should treat him like royalty!).<br />
<br />
BA's computer system had an amazing effect on all concerned, except most likely the people that wrote it. In my opinion the original developers should be there supporting the front-of-house staff. They should <i>feel</i> the pain that they have inflicted.<br />
<br />
I'm sure that you have similar stories to share, where computer systems have failed you miserably. Computer systems will of course fail, that's natural, but it is the fact that their degree of failure is considered acceptable that is the problem. Computer systems should not fail to the extent that they do. Your airline reservation system, online banking site, or whatever it is, it should be more reliable that it probably has been.<br />
<br />
The problem is that developers do not understand that building-in resilience to their software is more important than most other things. As my colleague, <a href="http://jonasboner.com/">Jonas Bonér</a> has stated many times, "without resiliency nothing else matters". He's so right. Why is it then that developers just don't get this?<br />
<br />
My answer to that is that many developers just see what they do as a job, and they don't really care about what they do. Putting that aside though, creating and then indeed managing distributed systems, a key requirement for resiliency, is harder than not; not hard, but harder and developers are lazy (btw: in case you don't realise this, I'm a developer!).<br />
<br />
We need systems that are resilient. We therefore need developers to care about resiliency. The more that developers care about resiliency, the more tools and technologies we'll see appearing in support of it. I strongly feel that it all starts with the developer though.<br />
<br />
I imagine a world where, given the inevitability of missing flight connections, I can wait in a queue for no longer than 10 minutes, be handled within another 10 minutes and then sleep off the tiredness and inconvenience of waiting another 24 hours for my next flight. The developer just needs to start caring in order for this to happen. Make he or she responsible for managing their system in production and they'll start caring, I guarantee it.<br />
<br />
Here's a language/tool agnostic starting point for you if you are a developer that cares enough to have read this far: <a href="http://www.reactivemanifesto.org/">The Reactive Manifesto.</a><br />
<br />
Thanks!Christopherhttp://www.blogger.com/profile/06860404627954267472noreply@blogger.com0tag:blogger.com,1999:blog-1413304312041976659.post-19010283940872703062016-05-22T12:46:00.000+10:002016-05-23T09:38:25.235+10:00Why we created an orchestration tool for youOne question I have had to answer a few times as the tech lead of <a href="http://www.lightbend.com/products/conductr">ConductR</a>, and I think it is a healthy question, is on why did <a href="http://www.lightbend.com/">Lightbend</a> create ConductR? This post is my personal attempt to describe the rationale for it two years ago, and why I think it is more relevant than ever.<br />
<br />
Back then we wanted a tool that was focused on making the deployment and management of applications and services built on Java, Scala, Akka and Play as easy as it could be. We wanted ConductR to be to operations what <a href="https://www.playframework.com/">Play</a> is to web application developers; a "batteries included" approach to deploying and managing reactive applications and services.<br />
<br />
Two years ago, there really wasn't anything else out there that we felt offered such a packaged approach to solving these new use-cases for operations people. The sentiment was that we had done a reasonable job with the <a href="http://www.reactivemanifesto.org/">Reactive Manifesto</a> at that point, and that we'd definitely engaged developers, but we were quickly going to arrive at a situation where operational people were going to find it a challenge to manage these new distributed applications and services. We also wanted something that had the reactive DNA.<br />
<br />
That's how it all started. So, what's changed, and why is ConductR relevant now?<br />
<br />
There are a number of players emerging in the orchestration space presently. This certainly validates our being a player in this space from a needs perspective. If you're happy to roll your own orchestration (which actually remains what we're up against in terms of competition, and this hasn't changed much in two years), then be prepared to have two people spend at least year tackling a problem that is harder than you think, and then realise that you have an operational cost in maintaining it. Atop of this, there's the risk to your company regarding those individuals leaving... is it sufficiently documented in terms of others taking over? Nobody has won in the orchestration space, but there's enough to choose from that will trump the business risk to your company of rolling your own. My advice here having been involved in designing and writing an orchestration tool (twice) is to not roll your own and focus on your core-business.<br />
<br />
While I personally think that the operational productivity culture that permeates through our design is still the single most important reason to consider ConductR, here are some other reasons:<br />
<br />
<ul>
<li>a means to manage configuration distinctly from your packaged artifact;</li>
<li>consolidated logging across many nodes;</li>
<li>a supervisory system whereby if your service(s) terminate unexpectedly then they are automatically restarted;</li>
<li>the ability to scale up and down with ease <b>and</b> with speed;</li>
<li>handling of network failures, in particular those that can lead to a split brain scenario;</li>
<li>automated seed node discovery when requiring more than one instance of your service so that they may share a cluster;</li>
<li>the ability to perform rolling updates of your services;</li>
<li>support for your services being monitored across a cluster; and</li>
<li>the ability to test your services locally prior to them being deployed.</li>
</ul>
<br />
<div>
Furthermore ConductR is the complete manifestation of the entire stack of technologies that we at Lightbend both contribute to and support. It is a great example of an Akka based distributed application that uses in particular, akka-cluster, akka-distributed-data and akka-streams/http. It is also tightly integrated with our Akka monitoring based instrumentation, and the monitoring story around events, tracing and metrics is going to get stronger. If you like our stack, you should feel good about the way ConductR has been put together.</div>
<div>
<br /></div>
<div>
We have programmed ConductR in the spirit of the Reactive Manifesto, with resiliency and elasticity being a particular focus. There is <b>no single point of failure</b> and our ability to scale out is holding up.</div>
<div>
<br /></div>
<div>
One last point: we use ConductR for our own production environment at Lightbend hosting our websites and sales/marketing services. With any product out there, you should always look for this trait. If a supplier is not dependent on their own technologies in terms of running their core business then beware; they can lose enthusiasm very quickly.</div>
<div>
<br /></div>
<div>
ConductR is as relevant as it ever was, and with its batteries-included approach for operations, I'm sure it'll become even more relevant as the industry moves toward deploying and managing microservices.</div>
<div>
<br /></div>
<div>
One last tidbit: ConductR is becoming a framework for Mesos/DCOS. Exciting times!</div>
<div>
<br /></div>
<div>
Thanks for reading this far!</div>
Christopherhttp://www.blogger.com/profile/06860404627954267472noreply@blogger.com0tag:blogger.com,1999:blog-1413304312041976659.post-32645077281595706912016-03-07T13:06:00.001+11:002016-03-07T13:16:15.893+11:00What the name "lightbend" means to meI thought that it'd be useful to share my personal perspective on the meaning of our company name change. Here are the contents of an email that I sent out to everyone within Lightbend, and which was warmly received.<br />
<br />
---<br />
<br />
Hi fellow lightbenders,<br />
<br />
I’m very excited about the <a href="http://www.lightbend.com/">Lightbend</a> name, and want to provide my view on what it means to me.<br />
<br />
About two years ago, I presented at YOW. YOW is a great conference with the characteristic that speakers get to talk to a cross section of our industry on three occasions: Melbourne, Brisbane and Sydney. One is therefore not preaching to the converted, but rather talking to what can be quite a hostile crowd!<br />
<br />
My first talk was to a few hundred people in Melbourne - apparently the most hostile of the three cities. About ten minutes into the talk I had that sinking feeling that I’d lost everyone. My talk was about Akka streams and the importance of back pressure. Lots of blank looks all around. An interesting aspect of YOW is that you are scored by the audience. You guessed it, my scores were low.<br />
<br />
Travelling up to Brisbane I felt that it was important to bring the talk back a bit. Instead of delving right into Akka streams, I felt that I should at least have a preamble around reactive streams and why we did that. The Brisbane talk went much better.<br />
<br />
However given the nature of the questions asked after my talk I felt that I could do even better. So, for Sydney, my preamble included a discussion on “why reactive”. This set the scene for the remainder of the talk and my Sydney scores reflected that.<br />
<br />
Coming away from YOW I realised how fringe Typesafe were - again this is two years ago. I certainly appreciated that we were not anything near mainstream, but really, we were on another planet compared to where the IT industry was at.<br />
<br />
Roll forward to today and you can see that we’ve come a long way. We have done so without deviating on our mission from an technical perspective. I would tell people that if you want to understand anything about our technical direction then simply read <a href="http://www.reactivemanifesto.org/">the Reactive Manifesto</a>. You'll then see our DNA blueprint; the very fabric of what we are. Taking that further and quoting <a href="http://jonasboner.com/">Jonas Boner</a>, “without resiliency, nothing else matters”. We have upheld the manifesto and, in particular resiliency, like nothing else matters.<br />
<br />
And now we are seeing the industry finally come our way. To highlight a few points, the industry received our new name well, it is excited about Lagom as a microservices framework for Java and the enterprise leading <a href="https://spring.io/blog/2016/02/09/reactive-spring">Spring framework is effectively adopting the reactive manifesto</a>.<br />
<br />
This is where the lightbend name kicks in for me.<br />
<br />
I see lightbend as the gravitational force that is bending the light beam representing the direction of the industry at large. Gravity bends light.<br />
<br />
To use my earlier analogy of Typesafe being on another planet, two years ago, we were light years away from where the industry was thinking. We are no longer. We have pulled the industry to the way we think software systems should be put together and managed.<br />
<br />
We are now at an interesting juncture. As the company expands as it needs to, it would be easy to compromise our technical mission in order to gain further traction. However it is now more important than ever to stay on mission.<br />
<br />
We need to be brave and continue to be bold. The industry doesn’t need more of the same; it needs more companies like us.<br />
<br />
Thanks for reading!<br />
<br />
Kind regards,<br />
ChristopherChristopherhttp://www.blogger.com/profile/06860404627954267472noreply@blogger.com2tag:blogger.com,1999:blog-1413304312041976659.post-42652780928543851972014-12-29T14:01:00.001+11:002014-12-30T05:56:50.031+11:00Where FP meets OO<p>A strong feature of Scala is the embracing of both Functional Programming (FP) and Object Orientation (OO). This was a very deliberate and early design decision of Scala in recognition of the strengths of both approaches over the decades. I hope to show you that utilising both approaches to developing software can work together.</p>
<p>Over the past two years of using Scala on a daily basis I’ve found myself adopting a predominantly FP approach to development, while embracing <a href="http://c2.com/cgi/wiki?AlanKaysDefinitionOfObjectOriented">true OO</a> in the form of Akka’s actors. What this boils down to is side-effect-free programming with functions focused on one purpose. Where state is required to be maintained, typically for IO style scenarios, then actors come to the rescue. Within those actors if there are more than 2 state transitions then I find myself using Akka’s FSM where upon it maintains the state through transitions. As a consequence, there are very few “var" declarations in my code, but I don’t get hung up on their usage where the code becomes clearer or there is a performance advantage in some critical section of code.</p>
<p>I can’t even remember the last time I used a lock of some type...</p>
<p>Thus my actor code looks something like this:</p>
<pre>object SomeActor {<br /> def props(): Props =<br /> ...<br /><br /> // My actor’s other pure functions, <br /> // perhaps forming the bulk of code. <br /> // The functions are typically private<br /> // to the package and therefore <br /> // available for tests within the<br /> // same package. <br />}<br /> </pre>
<pre>class SomeActor extends Actor {<br /> override def receive: Receive =<br /> ...<br /><br /> // Functions that break down the <br /> // receive handling calling upon the<br /> // pure functions of the companion and <br /> // possibly other traits<br />} </pre>
<p>What I find is that as I expand the companion object with pure functions, patterns of behaviour emerge and its functions are factored out into other traits which of course become re-usable and remain highly testable. Sometimes I form these behavioural abstractions ahead of creating the companion object, but more often than not it is the other way round. I’m big on continuous re-factoring and spend a lot of time attempting to get the functional abstractions right. This can mean that the functions are often re-written once the behavioural patterns emerge.<br /><br />So why then is the above representative of OO? Actors permit only message passing and are completely opaque to the outside of their instance. This is one of Alan Kay’s very early requirements of OO. Actors combined with Scala also mostly fit his following requirements (taken from <a href="http://c2.com/cgi/wiki?AlanKaysDefinitionOfObjectOriented">http://c2.com/cgi/wiki?AlanKaysDefinitionOfObjectOriented</a>):</p>
<ol>
<li>Everything is an object.</li>
<li>Objects communicate by sending and receiving messages (in terms of objects).</li>
<li>Objects have their own memory (in terms of objects).</li>
<li>Every object is an instance of a class (which must be an object).</li>
<li>The class holds the shared behavior for its instances (in the form of objects in a program list).</li>
<li>To eval a program list, control is passed to the first object and the remainder is treated as its message.</li>
</ol>
<p>Point 3 is a weak point of the JVM and one should be careful about message payloads remaining immutable, but at least with Scala, immutability is the default way of coding. It’d be great if actors themselves had their own address space. However this has never raised itself as a problem in my world.</p>
<p>Scala is one of the few languages that marries the world of FP and OO and thus does not need to “throw the baby out with the bathwater”. Many other languages force you to make a choice. That said, just like most marriages, there’s always the dominant party making the most sense, and that’d be FP!</p>Christopherhttp://www.blogger.com/profile/06860404627954267472noreply@blogger.com2tag:blogger.com,1999:blog-1413304312041976659.post-88765560826479771732014-01-01T15:04:00.001+11:002014-01-01T15:04:53.802+11:00Reflections on Java, JavaScript and Maven for 2013<p>About a year ago I made <a href="http://christopherhunt-software.blogspot.com.au/2013/01/java-javascript-and-maven-in-2013.html">some predictions on Java, JavaScript and Maven for 2013</a>. There has been some movement, so time to report back:</p>
<h2>Java</h2>
<p>Java 8 didn't quite make it as a GM release, but mid-March 2014 now appears to be the date. Java 8 has been available for playing with for some time during 2013 though.</p>
<p>I must confess to having been excited at the prospect of Java 8's lambda support a year ago, and I still think that what's coming is a great boost to the language. However I'm now squarely in the functional camp and, well, Java simply won't cut it. If you have an interest in functional programming, my personal recommendation is to move to a language designed for the job. Languages such as Scala which offer the best of the imperative and functional worlds are the ones to look at.</p>
<h2>JavaScript</h2>
<p>This one is mostly pinned to the release of Java 8 and Nashorn - DynJs hasn't really taken hold as I thought it might. So, may be March 2014 for this also.</p>
<p>Projects such as <a href="https://github.com/apigee/trireme#trireme">Trireme</a> are particularly interesting as they bring the Node API to Rhino. I suspect that projects like this can be adapted to Nashorn, and I also see that Nashorn may provide its own Node API implementation, <a href="http://mail.openjdk.java.net/pipermail/nashorn-dev/2013-June/001326.html">although the details on this are light</a>. No matter what happens regarding Nashorn and its Node offerings, I suspect the Trireme and Rhino combination will remain relevant for some time given their Java 6 focus.</p>
<h2>Maven</h2>
<p>Maven continues to be strong however my hope for an alternate DSL for the pom hasn't materialised… sort of… <a href="https://github.com/tesla/tesla-polyglot#description">Tesla polyglot</a> is ready for a release and offers Groovy, Ruby and Scala DSLs for Maven. I actually wrote the Scala dsl :-)</p>
<p>I suspect that Tesla polyglot will be released sometime during the first quarter of 2014.</p>
<h2>Conclusion</h2>
<p>I feel that my predictions were largely on track, although they certainly haven't materialised within the timeframe that I expected. There's been considerable progress across all three fronts though and I'd be very surprised if they haven't materialised by the first half of 2014.</p>Christopherhttp://www.blogger.com/profile/06860404627954267472noreply@blogger.com0tag:blogger.com,1999:blog-1413304312041976659.post-8084145233019646982013-05-01T14:27:00.001+10:002013-05-01T16:12:45.037+10:00Play-ing with WebJars<p>One of my responsibilities on the Play team is to enhance the JavaScript development experience. We will shortly be releasing a strategy document on what is coming for Play 2.3 in this regard. As a preview though, one of the things the strategy will be advocating for is the use of <a href="http://www.webjars.org">WebJars</a>. WebJars are JavaScript web libraries deployed to well known repositories including Maven Central. There are many popular JavaScript libraries already available as WebJars and the number is increasing.</p>
<h2>Preamble</h2>
<p>Why is there a need for WebJars? Managing the versions and dependencies of JavaScript libraries is just as important as for any other language. WebJars utilises familiar and established repositories instead of relying on newly introduced ones. I don't think that this can be understated; many organisations are already using <em>and hosting</em> Maven and Ivy based repositories so it makes sense to leverage them. To further support this, over 8 billion downloads occurred on Maven Central in 2012.</p>
<p>Why is dependency management important for JavaScript libraries? Some libraries are standalone of course but others are not. Many popular libraries have dependencies and it is the responsibility of the developer to source them and ensure they are available before the target library is sourced e.g. the popular <a href="http://twitter.github.io/bootstrap/">bootstrap</a> library depends on <a href="http://jquery.com/">jQuery</a>. Having a system that automatically manages the complexities of dependency management makes the JavaScript programmer more productive. WebJars enable such systems to be used.</p>
<p>The <a href="http://playframework.com">Play Framework</a> makes it easy to build web applications with Java & Scala. Play is based on a lightweight, stateless, web-friendly architecture. What we on the Play team have done as a first step is extend the work of my colleague, <a href="http://www.jamesward.com/">James Ward</a>, so that WebJars offers first class support of <a href="requirejs.org">requirejs</a> when used with Play. <a href="http://requirejs.org/">Requirejs</a> is a popular implementation of the <a href="https://github.com/amdjs/amdjs-api/wiki/AMD">AMD</a> specification - a means by which JavaScript applications can be modularised. The easiest way of thinking about AMD is that it is JavaScript's equivalent of package and import statements (or namespace and include statements depending on your preferences!).</p>
<h2>JavaScript and Play</h2>
<p>The first thing to state is that we want to make developing web applications in conjunction with Play as familiar as possible when it comes to authoring JavaScript. To use a WebJar the programmer declares its dependency in Play's Build.scala file. This file describes a Play project in a similar way that a <a href="http://maven.apache.org/">Maven</a> pom or <a href="https://npmjs.org/">NPM's</a> package.json would. Here's what a typical build file looks like using a variation of the <a href="https://github.com/angular/angular-seed">angular-seed</a> project <a href="https://github.com/typesafehub/angular-seed-play">extended for Play and WebJars</a>:</p>
<pre>object ApplicationBuild extends Build {</pre>
<pre>val appName = "angular-seed-play"<br />val appVersion = "1.0-SNAPSHOT"</pre>
<pre>val appDependencies = Seq(<br /> "org.webjars" % "angularjs" % "1.0.5",<br /> "org.webjars" % "requirejs" % "2.1.1",<br /> "org.webjars" % "webjars-play" % "2.1.0-1")</pre>
<pre>val main = play.Project(<br /> appName, appVersion, appDependencies)<br />}</pre>
<p>What is important to note is that the variable declared appDependencies specifies a list of WebJars that are required directly by the project. angularjs, requirejs and the webjars-play plugin are declared along with their versions. webjars-play actually depends on requirejs and so the above dependency declaration is not strictly required. However it is there to show that whatever requirejs version is declared by webjars-play, a different version can override it e.g. webjars-play depends on requirejs version 2.1.1 and so when requirejs version 2.2 is released then 2.2 can be specified above.</p>
<p>Knowledge of Scala is not required in order to declare dependencies. The above constitutes the total amount of Scala that the JavaScript programmer will be exposed to (unless they wish to delve into Scala which we would actively encourage of course!).</p>
<p>The WebJars website allows the easy selection of a WebJar and its version and then outputs the declaration required to download it for Play, SBT, Maven, Gradle and others.</p>
<p>The next requirement is to tell Play how WebJar assets are going to be resolved. Play's conf/routes file is used to do this. Here is the part of the routes file for the angular-seed-play project concerned specifically with WebJars:</p>
<pre># Obtain require.js with built-in knowledge of <br /># how webjars resources can be resolved</pre>
<pre>GET /lib/require.js controllers.WebJarAssets.requirejs</pre>
<pre># Enable webjar based resources to be returned<br />GET /webjars/*file controllers.WebJarAssets.at(file)</pre>
<p>The above tells Play to return a wrapper of requirejs whenever /lib/require.js is requested. This wrapper configures requirejs so that it knows how to resolve files hosted within WebJars. The /webjars declaration takes a file path and locates the corresponding resource from a WebJar.</p>
<p>The JavaScript programmer need do very little else to have Play deliver a JavaScript application.</p>
<h2>Requirejs usage</h2>
<p>Declaring the use of requirejs should look quite familiar. Here is a sample HTML snippet:</p>
<pre><script data-main="js/app" src="lib/require.js"></script></pre>
<p>Given the routes declaration from the previous section the above will bring in js/app.js after requirejs has been loaded. Note that Play must also understand the routing in terms of how to load resources from the js path:</p>
<pre>GET /js/*file controllers.Assets.at(path="/public/js", file)</pre>
<p>Here is what the angular-seed-play js/app.js file looks like with the less relevant bits removed:</p>
<pre>require([<br /> './controllers', <br /> './directives', <br /> './filters', <br /> './services', <br /> 'webjars!angular.js'], function(controllers) {</pre>
<pre> // Declare app level module which depends on filters, <br /> // and services<br /> ... </pre>
<pre>});</pre>
<p>The first few lines of the require statement declare dependencies on JavaScript files relative to the current one via the ./ convention. These files are required for the angular-seed project itself. The line of interest for this topic though is "webjars!angular.js". What this does is call upon the WebJars <a href="http://requirejs.org/docs/plugins.html">requirejs plugin</a> to load a file that is contained in a WebJar declared as a dependency.</p>
<p>…and that's about all there is to it.</p>
<h2>But wait, there's more...</h2>
<p>Let's say that bootstrap is required. Bootstrap has a dependency on jQuery. Ordinarily the JavaScript programmer is required to ensure that jQuery is loaded before bootstrap given its dependency. When using bootstrap's WebJar, jQuery is declared as a dependency such that:</p>
<pre>require(["webjars!bootstrap.js"], function () {<br /> ... </pre>
<pre>});</pre>
<p>...will automatically load jQuery. This is achieved given that bootstrap's WebJar declares how jQuery is to be located within a repository. Here's a snippet from the <a href="https://github.com/webjars/bootstrap/blob/master/pom.xml#L19">bootstrap WebJar pom.xml</a> file:</p>
<pre><dependencies><br /> <dependency><br /> <groupId>org.webjars</groupId><br /> <artifactId>jquery</artifactId><br /> <version>1.9.0</version><br /> </dependency><br /></dependencies></pre>
<p>The other thing that bootstrap's WebJar provides is a file named "webjars-requirejs.js" in a well-known location within the jar. Here are the contents of that file:</p>
<pre>requirejs.config({<br /> shim: {<br /> bootstrap: [ 'webjars!jquery.js' ]<br /> }<br />});</pre>
<p>The above declares to requirejs that whenever "webjars!bootstrap.js" is depended on, "webjars!jquery.js" will be loaded first. requirejs.config along with <a href="http://requirejs.org/docs/api.html#config-shim">the shim property is standard requirejs configuration</a>. In addition to requirejs behaviour, whenever a "webjars!" module is specified we strip off the "webjars!" prefix and the ".js" suffix and end up with a module name i.e. "bootstrap" in the case of "webjars!bootstrap.js". This module name is then looked up within the shim property of requirejs.config and, if found, the dependencies that are declared are loaded prior to bootstrap.</p>
<h2>One more thing...</h2>
<p>If there is a requirement to avoid using "webjars!" in JavaScript AMD dependencies then something like the following can be done:</p>
<pre>define("jquery", [ "webjars!jquery.js" ], function() {<br /> return $;<br />}); </pre>
<p>Thus any time that "jquery" is specified as a dependency its webjar will also be loaded e.g.:</p>
<pre>define([ "jquery" ], function($) {<br /> …<br />}); </pre>
<p>The above is just as if jQuery was declared to use AMD itself (<a href="http://blog.jquery.com/2011/11/03/jquery-1-7-released/">which it can do</a>).</p>
<p>We intend to enhance the JavaScript development experience further with the goal of making Play the #1 choice for web development.</p>
<p>Happy Play-ing with WebJars!</p>Christopherhttp://www.blogger.com/profile/06860404627954267472noreply@blogger.com8tag:blogger.com,1999:blog-1413304312041976659.post-42196317423722806592013-02-22T12:30:00.001+11:002013-02-22T12:30:42.601+11:00Hello Typesafe!<p><img title="logo.png" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg3IR4hH2sLTO0KPsC6yCpMRq6XjyZSY7oyHnm9znNsQRT9KLNJ2X-FVnR6b9sWXLoIGyQhTkC_9OnWvipx2w0WRKj-ROsj_eUo7hLPfi7vXKYG9Jo43OF-ZSr4ESBTz-ncUspv7pwuJGpT/?imgmax=800" alt="Logo" width="197" height="47" border="0" /></p>
<p>I'm thrilled to be able to announce that I am starting as a Senior Engineer with <a href="http://typesafe.com">Typesafe</a> this coming Monday. I'll be working on the <a href="http://typesafe.com/technology/play">Play</a> framework.</p>
<p>I'm excited to become involved in the Typesafe mission. In many ways Typesafe feels like an early SpringSource given its culture, focused people and established open-source projects.</p>
<p>I'll be based just south of Sydney (where I live) and work alongside my new colleague <a href="http://jazzy.id.au">James Roper</a>.</p>
<p>You'll be hearing more from me about why Play should be top of mind when considering the RESTful JSON based web service use case. Watch out <a href="http://nodejs.org">Node</a>, here we come!</p>Christopherhttp://www.blogger.com/profile/06860404627954267472noreply@blogger.com5tag:blogger.com,1999:blog-1413304312041976659.post-41005006849088504882013-01-08T08:56:00.001+11:002014-01-01T15:08:34.576+11:00Java, JavaScript and Maven in 2013<p>A couple of predictions for this year:</p>
<ul>
<li>Java will become cool again</li>
<li>JavaScript running on the JVM will be interesting</li>
<li>Maven will become cool</li>
</ul>
<h1>Java</h1>
<p>Java 8 will bring lambda functions into the fold and enhance its collections to support them. Functions as first-class objects are one of my favourite features of other languages such as JavaScript. What I like about them is the mainly the flow of writing code. With Java today there's a bit too much of a context switch when providing some type to handle lambda style behaviour. This can curb productivity.</p>
<p>It is a shame that automatic property access isn't being incorporated as I think that this would be the final nail that pushes aside a few other JVM languages.</p>
<h1>JavaScript on the JVM</h1>
<p>Both Nashorn and DynJs will change the landscape in terms of running JavaScript on the JVM. I'm personally quite impressed with Mozilla's Rhino and have always found it to be quite functional. Having something that builds off of our API learnings and simultaneously runs much faster will make the JVM a first class platform for running JavaScript. Watch out Node.js… I'm suspecting that they'll be a nice JVM based equivalent coming along.</p>
<h1>Maven</h1>
<p>It seems to me that people's main issue with Maven is the XML pom file format. Its about time that we had a better DSL for POMs. Imagine what a JavaScript POM DSL would look like… My belief is that moving away from XML will make Maven appeal to many current naysayers.</p>
<p> </p>
<p>Click here for my <a href="http://christopherhunt-software.blogspot.com.au/2014/01/reflections-on-java-javascript-and.html">reflections a year on.</a></p>Christopherhttp://www.blogger.com/profile/06860404627954267472noreply@blogger.com1tag:blogger.com,1999:blog-1413304312041976659.post-74902686615686968522012-03-03T17:29:00.001+11:002012-03-03T17:29:53.897+11:00Fast broadband == backup<p>Gosh, what a long time since I last blogged. Truth is I've moved over to R&D within VMware and I've been working on really interesting cloud stuff that has taken up all of my cycles. I'll be writing more about that soon.</p><p>There's a big debate here in Australia on the benefits of the <a href="http://www.nbnco.com.au/">National Broadband Network </a>(NBN). The opposition in government are asking for justification in terms of business case etc. etc. They're also arguing that the government's plan is rubbish stating that wireless is a better option. The arguments go on and I think are mostly inspired by the fact that the NBN isn't the opposition's idea.</p><p>I think that the biggest unstated benefit that will come out of the NBN is offsite backup. Yep, you heard it here, the boring business of backing up data.</p><p>My 1TB backup drive gave up the ghost a couple of weeks ago. Since then I've been thinking about buying another, but the thing is, I really want more offsite backup. I've got some offsite backup going in the form of other devices (iPhones, iPods etc.), and I'm even using our VMware Mozy backup solution for work stuff.</p><p>The things that are holding me back in terms of full offsite backup though is: (i) the ISP cost of uploading/downloading GBs of data; and (ii) the speed at which this can be done (both the initial upload and incremental uploads).</p><p>My view is that not only will the NBN deliver more speed (I'm only getting 3mbps at the mo), I think it'd be reasonable to expect larger download/upload quotas over time. Right now download quotas are pretty similar to existing ADSL plans but as more people suck the data down, the market will demand higher quotas. And that means that offsite backup should become a reality.</p><p>So the next time you're thinking about the cost of NBN here in Australia (presuming that you do!), think about how much your data is worth and then multiply that across the population!</p>Christopherhttp://www.blogger.com/profile/06860404627954267472noreply@blogger.com3tag:blogger.com,1999:blog-1413304312041976659.post-46969524385316949442011-09-08T07:40:00.001+10:002011-09-08T14:06:45.660+10:00Application developers should not write frameworks and toolkits (much)<p>On occasion I'm a little disappointed when I see developers create frameworks and toolkits. I understand that it is enjoyable to do so, but we, as application developers, should be majorly focused on the business needs of developing applications.</p><p>Quite often we think that xyz widget doesn't do exactly what is "required" and therefore pursue a path of days (weeks) writing frameworks and toolkits.</p><p>Let's not do this anymore.</p><p>Instead, every time you, as a developer, get the urge to write a framework or toolkit or even just a utility class, think about extending an existing open source project to meet your needs; or even starting a new open source project. For one thing, your requirements will most likely be challenged (which is a good thing). You will no doubt end up with something better than you could have created yourself (yes, that's hard to swallow, I know!). Importantly though, your organisation will benefit; the less "quirks" particular to an organisation the faster it is for them to ramp up resources and "Get Things Done".</p><p>Your organisation will also be able to share stuff between the silos. Imagine that.</p><p>I often hear, "but my organisation doesn't contribute to open source" (despite heavily using it). In this instance I always ask if the person in question has actually tried selling the business benefit (above) and attempted to make it happen. More often than not, the developer hasn't tried. Organisational behaviour is driven by the views of their employees and contractors. Organisations change and are changing.</p><p>I do believe that at the end of the day, it is better to suffer with an xyz widget that does most of what you need it to do than to re-invent the wheel. If you're an application developer then focus on the application, not frameworks and toolkits; unless you're contributing to open source projects.</p>Christopherhttp://www.blogger.com/profile/06860404627954267472noreply@blogger.com4tag:blogger.com,1999:blog-1413304312041976659.post-20310536069631198472011-08-11T13:28:00.001+10:002011-08-11T13:47:58.133+10:00Dependency Injection in Java - @Resource, @Inject or @Autowired?<p>The <a href="http://download.oracle.com/javaee/5/api/javax/annotation/Resource.html">@Resource documentation</a> states that the "name" attribute refers to "The JNDI name of the resource". Spring has overloaded this to mean that you can refer to bean identifiers… but as far as the Java spec goes, there's no contractual obligation for a resource name to refer to a bean id. @Resource referring to Spring bean ids is therefore quite Spring-specific.</p><p><a href="http://static.springsource.org/spring/docs/3.0.x/javadoc-api/org/springframework/beans/factory/annotation/Autowired.html">@Autowired</a> is Spring specific.</p><p><a href="http://download.oracle.com/javaee/6/api/index.html?javax/inject/Inject.html">@Inject</a> is not Spring specific and performs type-safe injection.</p><p>The other benefit you'll get from type-safe injection is the ability to safely re-factor within your favourite IDE, but I personally think that the contractual obligation mentioned above is more important.</p><p> </p>Christopherhttp://www.blogger.com/profile/06860404627954267472noreply@blogger.com0tag:blogger.com,1999:blog-1413304312041976659.post-51544420986099468352011-08-10T12:45:00.001+10:002011-08-10T13:52:27.133+10:00Versioning RESTful services<p>I've seen some comment out there regarding how to version RESTful services and so thought it about time to add in my own position on this.</p><p>I don't subscribe to <a href="http://blog.steveklabnik.com/2011/07/03/nobody-understands-rest-or-http.html">crafting your own MIME type</a> as these should refer to well-known entities. I also don't subscribe to passing a version number as an Accept param. Despite there being client/server compatibilities in this regard (Ruby), the developer is not forced to specify the version. I think it is useful to specify the version of a service to use.</p><p>I'm an advocate of using a /v1 path in the url of a RESTful service, but it should be noted that great care should be taken in exactly what that version number refers to. I've long subscribed to the major.minor[.maintenance[-build]] approach as per Maven and reasonably discussed at <a href="http://en.wikipedia.org/wiki/Version_number">Wikipedia</a>. In the case of URL versioning, only the <strong>major</strong> identification is used i.e. "1.0.2" becomes just "v1".</p><p>Major versions only change when there is an <a href="http://en.wikipedia.org/wiki/Version_number#Degree_of_compatibility">API-breaking</a> change that has been introduced. This should be <strong>very</strong> rare. API-breaking generally means quite a big shift in the approach an API is taking which is why I found it interesting that Google declare their Google Maps API <a href="http://code.google.com/apis/maps/documentation/javascript/basics.html#Versioning">will always be backwardly compatible</a>. To me that's effectively what held Microsoft back with Windows until Vista came along and started to drop backwards compatibility (Windows 7 of course is the evolution of that good decision).</p><p>So I say use /v[major] as a convention, but be diligent in the definition of what is "major".</p>Christopherhttp://www.blogger.com/profile/06860404627954267472noreply@blogger.com1tag:blogger.com,1999:blog-1413304312041976659.post-40979840943470786482011-07-22T22:36:00.001+10:002011-07-22T22:36:22.978+10:00LMAX and the benefits of keeping things in memory<p>I've now read Martin Fowler's review of the <a href="http://martinfowler.com/articles/lmax.html">LMAX architecture </a>and I feel good. This is very similar to what I've been doing for a long time with messaging (Spring Integration, Camel, AMQ, RMQ etc.).</p><p>LMAX is an event driven architecture. The Input Disruptor can be your favourite messaging broker. The Business Logic Processors are your message queue consumers. AMQP can certainly attain the goals described for the Output Disruptor (targeted consumers etc.).</p><p>The only thing "disruptive" about this article is that it introduces new terminology. I accept that there's probably a lot of optimisation going on, but from an architectural standpoint I don't see anything revolutionary.</p><p>Perhaps the most interesting points of the LMAX architecture are that everything is done in memory and that the processing of data is kept close to the data. These are things that we will all no doubt agree on being the main influence on performance.</p>Christopherhttp://www.blogger.com/profile/06860404627954267472noreply@blogger.com0tag:blogger.com,1999:blog-1413304312041976659.post-68644431894893847972011-06-26T17:31:00.001+10:002011-06-26T17:39:10.723+10:00RESTful JSON fixtures for testing using CouchDB<p> </p><p>I'm working in a distributed team at the moment where I don't have access to infrastructure available within the company's firewall. The application I'm working on has a nice <a href="http://en.wikipedia.org/wiki/Restful">RESTful</a> API with <a href="http://www.json.org/">JSON</a> payloads for its logic layer. As I'm mostly developing the presentation layer I needed a quick method of generating something that would simulate the logic layer.</p><p>One thing that I felt was important is that I shouldn't have to mutate any settings within my presentation application in order to work either against the simulated logic layer, or the real one. It is all too easy for such settings to creep into production and break things.</p><p>I had a few thoughts on how to achieve my objectives considering <a href="http://static.springsource.org/spring/docs/3.0.x/reference/mvc.html">Spring MVC,</a><a href="http://en.wikipedia.org/wiki/Jax-rs">JAX-RS</a> re-implementation using our existing interfaces (we're using JAX-RS in the real logic layer), writing a <a href="http://jetty.codehaus.org/jetty/">Jetty</a> based servlet handler and node.js. Then I remembered...</p><p><a href="http://couchdb.apache.org/">CouchDB</a> is a document oriented database that uses JSON as its Data Description Language (DDL). In addition CouchDB provides a RESTful JSON based API to access the database. This got me thinking that I could use sample responses from our real logic layer and enter them directly into the CouchDB database. It turns out you can using the <a href="http://wiki.apache.org/couchdb/HTTP_Bulk_Document_API">batch</a> interface!</p><p>I then thought that I needed to create a CouchDB service that could understand the parameters of the web service request I needed to make. These parameters are properties of my JSON objects that are otherwise unavailable to the URL format that CouchDB provides for querying. The answer here is to provide a <a href="http://wiki.apache.org/couchdb/HTTP_view_API">View</a>. Here's my view:</p><pre>function(doc) { emit(doc.surname, doc) }</pre><p>The surname is a property of my JSON object that I'll want to query.</p><p>I call the view the same name as my logic layer service. In my case the view's name is "getCustomerBySurname". If I query my service using CouchDB's URL convention I would issue something like:</p><pre>http://localhost:5984/customerdb/_design<br />/customerService-1.0/_view<br />/getCustomerBySurname?key=%22Hunt%22</pre><p>This yields a result like:</p><pre>{"total_rows":1,"offset":0,"rows":[<br /> {"id":"285b217949d54c29c4b27adf6d000da2","key":"Hunt","value":<br /> {"id":1,"surname":"Hunt","firstname":"Christopher"...</pre><blockquote><p>Note that I'm using a convention for versioning a resource so that I can evolve its API nicely in the future - this is quite important for RESTful requests as versioning isn't something supported by anything within HTTP</p></blockquote><p>I now have a web service that behaves the way the real service behaves. However, it doesn't look the same to the consumer for two reasons:</p><ul><li>the result contains meta information about the query that the consumer does not expect ("total_rows" etc.); and</li><li>the URL we provide is not of the same form as the real logic layer service.</li></ul><h2>Transforming the JSON result</h2><p>To transform the payload returned by CouchDB into something our consumer expects we create a <a href="http://guide.couchdb.org/draft/transforming.html">List</a> in CouchDB-speak. My list function looks like this:</p><pre>function (head, req) {<br /> provides('json', function() {<br /> var results = [];<br /> while (row = getRow()) { results.push(row.value); }<br /> send(JSON.stringify(results));<br /> });<br />}</pre><p>I've declared my list by the name of "rows". The function is focused on pulling out just the row.value property of each row and returning that back to the consumer. To access the view that I created and have it render through the list you provide a URL like this:</p><pre>http://localhost:5984/customerDB/_design/<br />customerService-1.0/_list/rows/<br />getCustomerBySurname?key=%22Hunt%22</pre><p>This now yields something like:</p><pre>[{"id":1,"surname":"Hunt","firstname":"Christopher"...</pre><p>...which is what my consumer expects.</p><h2>Transforming the url</h2><p>The url is transformed via <a href="http://httpd.apache.org/">Apache httpd</a> using a <a href="http://httpd.apache.org/docs/2.0/mod/mod_rewrite.html">RewriteRule</a>. In order to avoid <a href="http://en.wikipedia.org/wiki/Same_origin_policy">Same Origin Policy</a> we've had to proxy requests for our presentation component and its web services anyhow. Here is the re-write rule I ended up with:</p><pre>RewriteEngine On<br />RewriteOptions Inherit<br />RewriteRule ^/CustomerDB/services-1.0/rest/Customers?surname=(.*) http://localhost:5984/customerdb/_design/customerService-1.0/_view/getCustomerBySurname?key="$1" [QSA,P]</pre><p>... and that's it. I now have a means of using CouchDB to provide fixtures for the purposes of development.</p><h2>One last important thought...</h2><p>The really interesting thing about using CouchDB for the use-case I set out to solve is that I'm really starting to wonder whether the real logic layer should be written using CouchDB... it is so simple and powerful!</p><p> </p><p> </p>Christopherhttp://www.blogger.com/profile/06860404627954267472noreply@blogger.com0tag:blogger.com,1999:blog-1413304312041976659.post-60569815951943607852011-04-08T21:39:00.001+10:002011-04-08T21:42:22.379+10:00DAO coding paradigms can require some skill<p>"DAO coding paradigms can require some skill" is a sentence that presently <a href="http://en.wikipedia.org/wiki/Data_access_object">appears at Wikipedia</a> in the description of Data Access Objects. I find DAOs very useful and honing the skill is worth it. The article goes on to state a benefit being, "Improved efficiency and performance of the data layer since it is standard reusable software".</p><p>The majority of DAO implementations that I see tend to provide similar functionality to the <a href="http://en.wikipedia.org/wiki/Active_record">Active Record pattern</a> i.e. <a href="http://en.wikipedia.org/wiki/Create,_read,_update_and_delete">CRUD</a>. Unfortunately these DAOs do not achieve the benefit of improved efficiency and performance. Let's say I have a customer, with orders that relate to products. My use-case is that I want the customers along with the orders and their products. With a straight CRUD approach I tend to get the customers (c) and then I get the orders (o) and then I get the products (p). That's potentially c * o * p or more requests on the database over the network.</p><p>The method signatures of my DAOs are shaped by the service layer that drives them. Here are a few methods of one of my real-world DAOs to serve as an example:</p><pre>public void createSMSSubscription(<br /> String msn, String flightNo,<br /> String actualText, String receivedParsedText, <br /> Date scheduled, Date eventTime, <br /> AbstractSMSEvent subscriptionEvent) {</pre><pre><div>public Collection<Subscription> getActiveSubscriptions(<br /> AbstractRenderedFlight renderedFlight) {</div></pre><div><pre>public Collection<ReceivedSMSFlightSubscription><br /> getCurrentReceivedSubscriptions(<br /> String msn, Date receivedTime) {</pre></div><p>It may be that the service layer simply asks for a Customer to be created, but more often than not, the methods tend to be tailored to what the service layer wants to do e.g. "create an order for a customer that exists and only for products that exist", or "create an order for a customer that has not exceeded their credit limit" and so forth. I like to have my DAOs do as much as possible at the database with the objective of minimising the number of calls made to it. In short, my DAOs tend to comprise of fairly coarsely grained functions instead of finely grained CRUD style operations. I'll admit that it is a bit of an art in getting the granularity correct and that re-factoring commonly occurs, but the overriding goal is to reduce the IO between my logic layer and the data layer. Performance and efficiency then typically follow.</p><p>My DAOs are developed as the service layer evolves i.e. they are <strong>not</strong> developed before the service layer. As stated above I let the service layer determine what functionality the DAO should provide. If I've developed a service with the DAO development alongside, and then develop another service that wants more out of the DAO then I tend to done one of two things:</p><ul><li>parameterise the existing methods of a DAO such that their behaviour can be varied; or</li><li>create new methods on the DAO.</li></ul><p>One more thing while I'm thinking about DAOs... they're a great place to encapsulate persistence functionality of course. It used to be that we may have wanted to hide away the differences of dealing with multiple types of relational database. However we now find ourselves thinking about providing non-relational stores (NoSQL) as well. DAOs give you the freedom to choose the best persistence implementation to suit your application's requirements.</p><p>Long live the DAO!</p>Christopherhttp://www.blogger.com/profile/06860404627954267472noreply@blogger.com0tag:blogger.com,1999:blog-1413304312041976659.post-30390206676069873682011-04-03T02:22:00.000+10:002011-04-03T02:22:00.097+10:00Zoneinfo TZ v1.0.0 is now available!<p>My <a href="http://zoneinfo-tz.codehaus.org/">Zoneinfo TZ </a>library is now available up at the <a href="http://codehaus.org/">Codehaus</a>. V1.0.0 should make it to the central Maven repo shortly. The project has mailing lists, continuous integration, issue tracking and all of those things you'd come to expect.</p><p>Interestingly, the <a href="http://www.twinsun.com/tz/tz-link.htm">site</a> that hosts the zoneinfo files themselves has disappeared tonight. Fingers crossed that it'll come back soon!</p><p>I hope that this project proves useful to someone other than me! Enjoy.</p>Christopherhttp://www.blogger.com/profile/06860404627954267472noreply@blogger.com0tag:blogger.com,1999:blog-1413304312041976659.post-11757453499787173872011-03-02T09:18:00.001+11:002011-03-02T09:18:51.208+11:00GWT and GXT<p>Recently I've been doing a lot of development in <a href="http://code.google.com/webtoolkit/">GWT</a>. I have to say that, despite being a big fan of pure JavaScript <a href="http://en.wikipedia.org/wiki/Rich_Internet_application">RIA</a>, I'm liking GWT a lot. It has been a couple of years since I touched GWT and it has come a long way. I particularly like the <a href="http://code.google.com/webtoolkit/doc/trunk/DevGuideMvpActivitiesAndPlaces.html">Activities and Places</a> API, although there's a bit of a learning curve there.</p><p>One thing I don't particularly like though is <a href="http://www.sencha.com/products/extgwt/">Sencha's EXT-GWT</a>, or GXT as it is known. For those that don't know, GXT provides an abundance of widgets including a fully featured Grid. This is of course highly enticing, but what I've found is that GXT appears to be an all-or-nothing proposition. GXT is more akin to a framework than a toolkit because it effects how you lay out your code. What isn't clear to me is why it has its own layout architecture and its own event architecture. That's just confusing!</p><p>I'm starting to think that GXT plugged a few gaps in GWT 1.x and is becoming increasing less relevant. GWT 2.x deprecates a lot of GXT; even GWT 1.6 introduced a new event model.</p><p>Don't even get me started on GXT's Grid separation of concerns; its one of those widgets that likes to influence how the data transfer objects look. Plain wrong. The whole loader concept is terribly confusing as well.</p><p>I feel that if GWT's TableCell was as fully featured as GXT's Grid then no one would probably use GXT.</p><p>Perhaps I'm being a little harsh though. GXT widgets present nicely. It is just that they don't just "slot in"; you've got to buy into all of GXT to use a little of it.</p>Christopherhttp://www.blogger.com/profile/06860404627954267472noreply@blogger.com2tag:blogger.com,1999:blog-1413304312041976659.post-31753985984893429072011-01-13T10:13:00.001+11:002011-01-13T10:13:50.755+11:00I have a new job<p><img title="spring09_logo.png" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiMIE2mS1h6iAmTKBAI2NQ2MlJGr2-9hU4ebirkkFVSAK-22OtXjR5Y-uLFBuJ-JDrXhXNd7ODvvBFrUaH2FyqOnW7CFdpT1joHtr9tH7s04S1tgzdT2AelodDvPIX0seikm22BS_nzOKHu/?imgmax=800" border="0" alt="spring09_logo.png" width="240" height="50" /></p><p>I now have a job with VMware starting January 24th and I'm excited! My role is as a <a href="http://www.springsource.com/products/cloud-application-platform">vFabric</a> consulting software engineer within the professional services division .</p><p>Over the past 12 years I've been an independent contracting developer/architecture/designer and launched quite a few of my own products. I suppose one of the initial drivers to go contracting for me back then was that it was lucrative to do so. I now feel that the gap in remuneration between going permanent and contracting has diminished.</p><p>Another incredibly important factor was my family. Contracting can be great but the cash flow can be stressful. With 4 kids I needed more stability.</p><p>Finally, I wanted to work with more like-minded people. During my contracting years I've met some truly inspirational people along the way; but I want to meet more! This can be difficult as a contractor as you tend not to stay around long enough to work on the strategic projects and with an organisation's best people.</p><p>So, back in May last year I decided to look around for some permanent work. I choose some companies with <a href="http://en.wikipedia.org/wiki/SpringSource">SpringSource</a> being one of them. I thought, if I go for a permanent job then who would I really want to work for? It is important to "tow the corporate line" of course and so I needed to ensure that this would be a no-brainer. I'm not very good at lying!</p><p>I've been using the Spring framework for quite some time and became very interested in the work of some of the SpringSource people; namely <a href="http://www.springone2gx.com/conference/speaker/rod_johnson">Rod Johnson</a> and <a href="http://www.springone2gx.com/conference/speaker/ben_alex">Ben Alex</a>. Towing the line for SpringSource would not be a difficult proposition... I contacted SpringSource and ended up meeting with <a href="http://au.linkedin.com/pub/guy-baldwin/0/51a/764">Guy Baldwin</a> and the ball then started to roll. I instantly liked Guy and his technical knowledge along with his amiable personality makes him a great face of VMware/SpringSource.</p><p>'better had just get one thing straight: VMware acquired SpringSource in 2009 and so SpringSource is now a division of VMware. I'll refer to SpringSource as VMware for the rest of this blog ('hope that is politically correct!).</p><p>vFabric is the name given to VMware's Cloud Application Platform, meaning the whole stack of technologies that enable applications to live in a cloud environment. Technically speaking the cloud may not be the actual cloud as in the internet, but live behind the corporate firewall; it doesn't really matter though. What the platform is really about is enabling applications to be scaled in order to meet unanticipated demand.</p><p>My specific role is a first here in Asia/Pac. VMware have a vFabric consulting presence in the US and I'm their first consultant for them here in our region. It is exciting.</p><p>I'll continue to post my own views on software development here but needless to say, they'll be my views and not necessarily VMware's!</p>Christopherhttp://www.blogger.com/profile/06860404627954267472noreply@blogger.com4