Introducing "standard streamed commands and events", or "stdce"

This blog introduces a stream-based command/event programming model that can be applied to many programming languages. Having an event-driven approach to programming like this helps achieve resiliency and responsiveness. Resiliency should speak for itself, and responsiveness often leads to an improved user experience.

Since writing up on Event-drive Finite State Machines a couple of years back, colleagues and I have worked on edfsm and the flip-flop protocol. "edfsm" is a Rust implementation of that previous write-up, and the flip-flop protocol is a half-duplex protocol for conveying commands and events. Commands and events are the common denominators to all that I've just referred to and form the foundation of what I'd like to introduce here today: the formality of a programming model that has surfaced for me, and one that can be used to shape many types of program, including those solved presently by "function-as-a-service" (FaaS). That's the last mention of FaaS here though; I don't want this idea to be constrained to this one, albeit popular approach to programming!

The approach I wish to formalise uses standard streams (stdio) and is essentially this:

• stdin is used to receive a continuous stream of commands
• stdout is used to emit a continuous stream of events where events can be of type "logged" or "ephemeral"

...and that's it.

The approach enables long-running event-driven programs to be written in virtually any programming language that supports stdio, and most programming languages do. Furthermore, the approach conveys the notion that events can be stored in an event log (logged), or that they may come and go where their loss is insignificant (ephemeral).

Pipelines can also be achieved given stdio. For example, a program hosted in an OCI container might be communicated with via Unix Domain Sockets producing and consuming with the program's stdio.

Using stdio to receive a stream of commands and emit a stream of events may not even be so novel. I'd be interested to learn of similar efforts and make comparisons. That said, this particular blog isn't going to fully formalise the approach either. That's better served with examples. But, I would like to capture a name for the approach and introduce it at a high level. To that end, I'm going to name this approach "standard streamed commands and events", or "stdce" for short.

Based on the work I've done in this space already, here are some commands and events that may be required. There will probably be more!

Commands

A command is delivered on stdin as JSON using the following form:

{

  "command": "<some-command>"

  "args": {

    "<some-arg>": <some-arg-value>

  }

}

The following commands are envisaged:

• Configure
• HandleRecoveryEvent
• WaitForEvents
• HandleEvent

The Configure command is the first command received and provides any secrets and other configurations that the program may require. I've previously used secrets to decrypt command payloads and encrypt event payloads.

HandleRecoveryEvent is sent for each event sourced from an associated log i.e. previously "logged" events. A program may reconstitute its state from these recovery events. A WaitForEvents command will be sent when all recovery events have been sent.

HandleEvent is sent for each event that occurs post-recovery. Events may be sourced from many topics including those produced by other processes.

Events

An event is a JSON output to stdout using the following form:

{

  "event": "<some-event>"

  "type": "<event-type>"

  "args": {

    "<some-arg>": <some-arg-value>

  }

}

Any event that the program wishes to emit can be conveyed using an event type of "Logged" or "Ephemeral".

Configuration

Not mentioned so far, but of course, a program will need to provide a configuration in terms of "topics" to subscribe to namespaces and other things. This information can be communicated outside of stdce and is left as an implementation concern.

Next steps

This blog is a starting point, and somewhere for me to just get the idea down and name it. I'd like to get some feedback and then expand by fleshing out a project on GitHub. Hopefully, more to come soon then...