4P: a four-protocol API for synthetic filesystems

I've been implementing a lot of synthetic filesystems lately, and it struck me that (at least in the Plan 9 world, but it also seems to hold true for many Linux synthetic file systems) there is a lot of boilerplate gunk in synthetic file systems which is largely unnecessary -- particularly for control interface synthetic file systems (as opposed to normal disk access file systems or some sort of data-exchange file system). For this I'm talking more about interfaces like /proc.

In any case, it seems to me what things largely boil down two is two basic operations, get and put, with perhaps two other protocol elements for flushing and outstanding request and reporting errors. By and large this is the same conclusion that the Octopus protocol [1] folks came to (although they have more complexity and operations than I think are necessary), and its also the fundamental observation underlying web-style RESTful transactions.

The advantage of a minimal interface for synthetic control file systems is that it should allow you to keep the server implementation extremely simple and it will minimize the over-the-wire transactions necessary to complete a given operation (which for many control interfaces is even more critical due to the synchronous nature of walk, open, read, close or walk, open, write, close operations -- particularly for large latency network links). This last point was, I believe the primary motivation for the Octopus folks.

So, punting on security for the moment, the basic principle is to have the 4 protocol operations (Get, Put, Flush, Error -- hence 4P). They follow the same basic protocol convention of 9P with a basic header of size[4] op[1] tag[2]. We ditch fids and the state associated with them for just sending the full path on every operation -- the basic observation being that in most synthetic control file systems we tend to open, read/write, and then close the file -- so there is no reason to have the file held open, and path resolution is simple enough to obviate the need for fid processing/tracking. The basic operations are similarly simplified:

size[4] TPut tag[2] flag[s] path[s] meta[s] data[s]
size[4] RPut tag[2]

Put ends up assuming the job of creation as well as writing data. RESTful semantics are assumed, so individual operations are treated as atomic and whole-file operations -- there is no partial update in the protocol, so no offsets. meta[s] is actually a subset of the typical stat style meta-data, as control systems typically only have user, group, and perm style metadata. flag[s] allows some extended operations and semantic features allow exclusive creation (don't create if it's already there), blocking (to wait for it to not be there before putting data we have), append operation (add to the end instead of just replacing), as well as specification of meta-data only put operations (which might change owner but not data for instance). Using a string for the flags allows file-specific flags (which might lead to nightmares, but I'm leaving it in for now), and partial support of flags is of course allowed (returning an error on an unsupported flag).

size[4] TGet tag[2] flag[s] path[s] nmsg[4] count[4]
size[4] RGet tag[2] meta[s] data[s] more[1]

Get has a similar simplified nature, although it does allow specification of partial reads (although always from the top of the
file) by using count[4]. Since gets can also be streaming (multiple reply messages, get allows you to specify the maximum
number of reply messages you wish to receive relating to this transaction) - the response more field lets the client know when there are more packets coming related to this transaction on the same tag. Get also accommodates flags allowing the specification of advanced functionality including removing the file after getting (linda style), truncating the file after getting, waiting for an update to be put into the file before processing a get (server relative time), only retrieving data, only retrieving metadata, and specification of streaming (allow up to nmesg[4] return messages on updates to the file).

From a client perspective we could build these as either a 9P gateway or a VFS file system (as well as building a client library for tighter binding with an application or application(s)). From a server perspective, we could build servers in a similar fashion to the golang http servers, allowing handlers to be registered for elements, using regular expressions for allowing parameters to be encoded into paths. To enhance modular construction, we could allow for hierarchical representation of the handler tables which should also make search more efficient.

Using such a system it would be possible to construct a hello world file server in a few lines of code, with more complicated dynamic control file systems able to be constructed with not too much additional effort since the system takes care of managing the path hierarchy for you. Its not intended for such a system to be used for typical storage file systems or anyplace where large amounts of data need to be processed or accessed with varying degrees of granularity -- but for command/control file systems this should provide a more optimal solution both in terms of time to develop as well as network latency and efficiency.

A big motivation here as well as this takes some of the state out of the 9P protocol, which might allow more complicated distributed dynamic command/control frameworks. I imagine this might be particularly useful for xcpu style distributed execution environments -- which is where we intend to play with the approach.

[1] Octopus Protocol: http://lsub.org/ls/export/op.pdf