Grave Robbers From Outer Space

Standing Desk

2011-02-07T18:54:00.004-06:00

Since building standing desks seem to be all the rage, I decided to put one together for work. I used an Ikea Fredrik as the base, but only used the two shelf pieces and discarded the main desk piece as I wanted it to fit in a corner. The Fredrik has multiple slots along the side allowing you to position the shelves wherever you'd like. One thing I would caution is decide once where you want the stuff as pulling it apart and reassembling is a bit of a chore and you have to built it from the bottom up. It's got quite a nice solution for cable routing and hiding power strips. I also used the Fredrik Accessory Pack mostly so I could add cup holders to the desk, but the hooks came in quite handy for wire routing and for hanging my backpack. The official instructions actually have variations which seem appropriate for standing desks. One thing I did notice with the way mine is configured is it a bit shaky if I type too hard. The vibration isn't pronounced, but its enough to slightly shake the LCD on my laptop.

I attached a 3M keyboard tray to the top shelf, but had to use some nylon spacers to attach it since there is a metal support beam running down the center of the shelf. I used two cheap LCD arms for the extra monitors -- although these tend to tilt slightly with the way I have them mounted. They also need some wood blocks added under the shelf so they have the right thickness to attach to. To top it off, I bought a cheap drafting chair for days when I feel a bit tired of standing -- but I went for one that looked uncomfortable so that I wouldn't be tempted to sit all that often.

I've been using it for a couple of weeks now and seem to be able to stand most of the day while being productive.

IBM Centennial Film: 100 X 100 - A century of achievements that have cha...

2011-02-01T14:27:00.001-06:00

Multi-pipes

2010-10-05T12:19:00.005-05:00

Here's our poster we presented at OSDI last night on multi-pipes. Will be giving a short talk on them at IWP9 which you should be able to view next week here.

4P: a four-protocol API for synthetic filesystems

2010-08-27T16:24:00.002-05:00

I've been implementing a lot of synthetic filesystems lately, and it struck me that (at least in the Plan 9 world, but it also seems to hold true for many Linux synthetic file systems) there is a lot of boilerplate gunk in synthetic file systems which is largely unnecessary -- particularly for control interface synthetic file systems (as opposed to normal disk access file systems or some sort of data-exchange file system). For this I'm talking more about interfaces like /proc.

In any case, it seems to me what things largely boil down two is two basic operations, get and put, with perhaps two other protocol elements for flushing and outstanding request and reporting errors. By and large this is the same conclusion that the Octopus protocol [1] folks came to (although they have more complexity and operations than I think are necessary), and its also the fundamental observation underlying web-style RESTful transactions.

The advantage of a minimal interface for synthetic control file systems is that it should allow you to keep the server implementation extremely simple and it will minimize the over-the-wire transactions necessary to complete a given operation (which for many control interfaces is even more critical due to the synchronous nature of walk, open, read, close or walk, open, write, close operations -- particularly for large latency network links). This last point was, I believe the primary motivation for the Octopus folks.

So, punting on security for the moment, the basic principle is to have the 4 protocol operations (Get, Put, Flush, Error -- hence 4P). They follow the same basic protocol convention of 9P with a basic header of size[4] op[1] tag[2]. We ditch fids and the state associated with them for just sending the full path on every operation -- the basic observation being that in most synthetic control file systems we tend to open, read/write, and then close the file -- so there is no reason to have the file held open, and path resolution is simple enough to obviate the need for fid processing/tracking. The basic operations are similarly simplified:

size[4] TPut tag[2] flag[s] path[s] meta[s] data[s]
size[4] RPut tag[2]

Put ends up assuming the job of creation as well as writing data. RESTful semantics are assumed, so individual operations are treated as atomic and whole-file operations -- there is no partial update in the protocol, so no offsets. meta[s] is actually a subset of the typical stat style meta-data, as control systems typically only have user, group, and perm style metadata. flag[s] allows some extended operations and semantic features allow exclusive creation (don't create if it's already there), blocking (to wait for it to not be there before putting data we have), append operation (add to the end instead of just replacing), as well as specification of meta-data only put operations (which might change owner but not data for instance). Using a string for the flags allows file-specific flags (which might lead to nightmares, but I'm leaving it in for now), and partial support of flags is of course allowed (returning an error on an unsupported flag).

size[4] TGet tag[2] flag[s] path[s] nmsg[4] count[4]
size[4] RGet tag[2] meta[s] data[s] more[1]

Get has a similar simplified nature, although it does allow specification of partial reads (although always from the top of the
file) by using count[4]. Since gets can also be streaming (multiple reply messages, get allows you to specify the maximum
number of reply messages you wish to receive relating to this transaction) - the response more field lets the client know when there are more packets coming related to this transaction on the same tag. Get also accommodates flags allowing the specification of advanced functionality including removing the file after getting (linda style), truncating the file after getting, waiting for an update to be put into the file before processing a get (server relative time), only retrieving data, only retrieving metadata, and specification of streaming (allow up to nmesg[4] return messages on updates to the file).

From a client perspective we could build these as either a 9P gateway or a VFS file system (as well as building a client library for tighter binding with an application or application(s)). From a server perspective, we could build servers in a similar fashion to the golang http servers, allowing handlers to be registered for elements, using regular expressions for allowing parameters to be encoded into paths. To enhance modular construction, we could allow for hierarchical representation of the handler tables which should also make search more efficient.

Using such a system it would be possible to construct a hello world file server in a few lines of code, with more complicated dynamic control file systems able to be constructed with not too much additional effort since the system takes care of managing the path hierarchy for you. Its not intended for such a system to be used for typical storage file systems or anyplace where large amounts of data need to be processed or accessed with varying degrees of granularity -- but for command/control file systems this should provide a more optimal solution both in terms of time to develop as well as network latency and efficiency.

A big motivation here as well as this takes some of the state out of the 9P protocol, which might allow more complicated distributed dynamic command/control frameworks. I imagine this might be particularly useful for xcpu style distributed execution environments -- which is where we intend to play with the approach.

[1] Octopus Protocol: http://lsub.org/ls/export/op.pdf

Linux 9P Trace and Walkthrough

2010-01-27T21:34:00.002-06:00

Unfortunately video didn't come out.

9P Overview Slides

2010-01-27T15:00:00.005-06:00

MP4 Video Link or Livestream below:

Watch live streaming video from graverobbers at livestream.com

International Workshop on Plan9 - live streaming video powered by Livestream

2009-10-23T09:28:00.000-05:00

International Workshop on Plan9 - live streaming video powered by Livestream

Grande

2009-10-06T09:21:00.003-05:00

As a quick experiment the other day, I wrote a file-system back-end for Venti which serves the Venti protocol, but instead of using Indexes/Arenas/etc. for the back-end, it creates a hierarchical directory structure similar to Git with a file per score. This was done in the context of some other experiments (which I will post about later when they are complete), but I figured this may useful in of itself. The code is currently available in my vdiskfs branch of Plan 9 Ports on bitbucket (http://bitbucket.org/ericvh/vdiskfs/src/tip/src/cmd/venti/grande.c). It is very much a prototype, so not very robust -- but feel free to use it or build on it if you think it is useful to you.

acme-js

2009-08-04T10:53:00.002-05:00

I wonder how hard it would be to craft a web 2.0 ACME -- the basic widget sets are all there more or less, the new portal examples from the prototype JS libraries would be most of the work, just need handles to allow stretching the portals and a bit more interactivity in the title bars. Chording may be a bit of an issue, but according to the W3C spec I should be able to detect events for all three buttons.

Of course, there is more to it than just the interface. The back-end would be a file system providing both the traditional ACME synthetic file server and a separate directory (or perhaps view) with the necessary UI components (as JS and whatever). I was thinking a third view might be necessary for interaction with other users, but perhaps that could all be managed from the system view.

stats

2009-07-16T09:27:00.001-05:00

For profiling and regression tests it would be nice to log snapshots of certain system performance counters (/dev/sysstat, /dev/swap, etc.) before and after the kernel of the benchmark (which may not be the entire application). It would also be nice to be able to enable periodic logging (although that would interfere with many benchmarks -- particularly OS noise). Periodic logging interference could be mitigated by placing the log process on a different core than the application kernel core (when possible).

I think the general idea would be to have a file in the namespace which can be written to to either snapshot the stats or start periodic logging on a particular core. The file could then be read to get the snapshot data in some convienient format. If this was all done on a single open that may even allow for several simultaneous users -- although the specific scenario that we are using probably wouldn't require that.

Tree and Torus Woes

2009-07-08T11:15:00.003-05:00

Some of my modifications to cpu (specifically chdiring after backmounting the initiator to allow for a poor-man's desktop extension model) seems to have tickled a big bug on the CPU nodes -- namely the torus starts getting errors even though the torus isn't being used (by anything). Could be a sign of major memory corruption, but I doubt the cpu application itself could be the cause. More likely the backmount which becomes the CWD of the compute node starts issuing lots of 9P requests over the tree and things fall down. Of course it could just be a slightly different binary tickling the system in a slightly different way and producing a corner-case error condition that scrambles memory. At least it seems to be reproducable.

CPU, Desktop Extensions, and Caches

2009-07-08T11:12:00.003-05:00

Okay - having the default be mounting our home directory seems to be a really bad idea. ftq- which writes one line at a time takes f-o-r-e-v-e-r to complete just writing out the data. We need a custom cfs that does aggressive cache and also does some write-back caching as well for this to even remotely work.

Of course that will fly in the face of doing this from all cpu nodes to a single file.

This turns out to be a nice driver, doesn't it? So what we want is a single cache running on the IO node and the cpu nodes mounting that. That could be a matter of switching to the chaining cpu method (of course that would require more modifications to cpu.c).

The other thing I've been toying with is the idea of being able to pass a setup script or namespace script to cpu in the same manner than xcpu allows. This would allow us to establish a default namespace, including mounting relevant file servers and/or setting up cache servers.

The other side of this is that this work will all be completely obviated once we have the execution model in place -- but it is useful to have this naive approach as a point of comparison to justify the execution model.

HARE FastOS Workshop Talk Video Available

2009-06-29T15:16:00.004-05:00

The video podcast of my talk on HARE for the FastOS June 2009 Workshop is now available for download or from iTunes. Slides are available separately here.

On regressions and progress

2009-06-25T16:35:00.002-05:00

Initial take will just use the single threaded ftq for regression, we still need a regression script to organize results, but this should be relatively straightforward since all the version info is availabel in the #P device. Not clear how we get md5sum of the binary unfortunately, we can make certain assumptions based on surveyor. To keep things simple, we'll start with hardcoded version and then generalize.

bugs in lat_ctx may be timing related, potentially complicated by the virtual machine. Need to get criswell rebooted and try out there to make sure its not something stupid tripping things up. Be good to have ftq and 9bench numbers for a typical x86 machine to calibrate against anyways.

Nope FP errors happen on criswell as well, something is buggered in the timing code when the measurements are too small.

Okay, not a whole lot of progress today. Futz'd a bit with the lmbench code to see what works and what doesn't. It will be a nice set of numbers at the end of the day and we have the x86 code to compare against.

Since criswell is running the same hardware as the box next to it I should be able to get numbers for Plan 9 vesus Linux easily enough and we can use these as baselines for the normal Plan 9 regressions. We'll need to get the kittyhawk and/or zeptos code base running and try either lmbench or our modified versions there.

Of course, the thing I was suppose to be doing today was getting a top-level script together which runs on the desktop and initiates data collection (initially only of ftq) and stores results in a hierarchy based on kernel configuration (as read by '#P'/version and md5sum 9bgp.elf). I suppose I may still be able to pull those basics together this evening.

Regressions

2009-06-22T15:56:00.003-05:00

Its become clear in the process of the HARE work that we really need a regular regression suite to verify functionality as well as track performance over the course of development. It strikes me that whatever we come up with may be generally useful for the Plan 9 kernel since it seems we don't really have any quantitative tracking of kernel performance.

Seems like one of the first things we need is better information about the kernel encoded in the binary. We've got the date in KERNDATE of when it was build, but little information about where and when it was built (and also, by who).  This seems increasingly important in an environment where we aren't just building out of a single directory. For my purposes, some sort of tree identifier such as a venti score or a git HEAD would be useful to encode to track exactly which version of the kernel we are testing. Eventually it seems like it would be a good idea to have a unique id for the build tools and root file system as well.

Regressions will be stored in a temporal directory hierarcy (similar to yesterday), then a directory with the $OBJTYPE filled a directory labeled by the md5sum of the kernel binary at the leaf. This directory will contain an info file with extracted meta-data about the kernel and a numbered subdirectory per run collected. Each of the numbered subdirectories will contain a console output file, and a test output file per test (named by the test). There will also be an info file with any relevant information about the run (in the case of BG/P this will be the personality and bump-id).

The regression binaries will be included with the kernel as part of the initial ramdisk to make things simple at first. The script will be modular and run from the front-end node, requiring nodes be able to boot to the point where one could cpu into them.
This will also require us to be able to determine the assigned IP address of the IO Node(s) automatically. It makes sense to go for 128-node configurations with 2 IO nodes so we can test the Ethernet bandwidth between IO nodes and not just to the front end.

The idea right now is to have a set of 5 initial performance regressions covering computation, sparse memory access, OS interference, file system access, and network access using a typical 4-way SMP configuration. Eventually we may broaden these tests to cover a range of configurations including uniprocessor and large-page. The general idea is to have a short regression which runs around a minute or so for simple sanity check of the system at boot-up and have a longer regression which runs on a daily schedule and does a more comprehensive evaluation.

For the longer test one option might be to measure the time it takes to build the entire system from scratch (kernel build or kernel+applications) on every Blue Gene node hitting a single file server for source files and storing results to a ramdisk. A variation on this would be to develop a parallel build system in which all 128 nodes collaborated on compiling the system.

Another idea I am considering is the use of linuxemu to run Linux benchmarks under Plan 9 to both measure the kernel and the emulation infrastructure. Candidates would include lmbench, postmark, dbench, and netperf.

The initial focus is on gathering results, but the idea is one we have started collecting data to allow web-based visualization of the results tracked over time so we can monitor the general health and performance of the system, As use cases materialize for testing we'll add them to the matrix.

FastOS Podcast Launched

2009-06-18T12:13:00.006-05:00

I went through and worked out the creation of a video podcast including setting up the right iTunes syndication. I'll be posting all the FastOS talks (including the HARE/Plan 9 one) eventually. They (and the slides) are available on the FastOS site (http://www.fastos2.org) and via iTunes

Its fairly easy, assuming you have a good mic for the speakers. I may try to go and clean up the old IWP9 talks we have video of, but as a community it would really be a good idea to start making video podcasts of talks, tutorials, and whatnot. All the kids are doing it these days after all. I'm thinking of going back and capturing all of my past talks that I still have the slides for.

Plan 4?

2009-06-18T12:13:00.001-05:00

Perhaps instead of changing names or increasing numbers with Plan 9, we should reduce numbers in subsequent versions since we strive to have less code and complexity (unlike certain other operating systems).

Of course, there is the problem that all the other Plans did fail to enslave humanity. Still, there would be something nice to a Plan 4 which was more compact and simple, and it could use a reduced operating protocol 4P for communication. Of course, counting down from 9, we have a short window of opportunity to succeed.

Inferno ACME BG/P Environment

2009-05-28T14:50:00.001-05:00

Have everything setup now to be able to launch and mount a remote inferno and then interact with the host using its devcmd and file system (and its network).

This lets me deal with BG/P (and cobalt and telneting to the nodes) completely inside ACME -- except the Inferno telnet (inside ACME anyways) is spewing CRs and not echoing my input.  Blech.

ACME Trivia

2009-05-28T09:44:00.001-05:00

So Feedkey allows me to setup ACME to prompt for Factotum challenges -- but how do I get Feedkey to start every time I start ACME?  It doesn't show up in the dump, so its not as clear as autostarting other aspects of the system.

export over ssh stdio

2009-05-27T17:48:00.004-05:00

So, I've modified the lcmd stuff a bit and am able to mount a remote inferno instance that I start with an os ssh:

(local script)

#!/dis/sh

load std expr string
args := $*
s := sh -c ${quote $"args}
mkdir -p /tmp/lpipe
bind '#|' /tmp/lpipe
echo os
os ssh $remote /home/ericvh/inferno/Linux/power/bin/emu -I -r /home/ericvh/inferno /dis/sshexport /tmp/lpipe/data1 &
echo shell
mount -A /tmp/lpipe/data /n/remote
/dis/sh
echo halt > /n/remote/dev/sysctl
unmount /n/remote

(remote script)

#!/dis/sh
load std
or {ftest -e /net/cs} {ndb/cs}
bind -c '#U*' /n/local
bind -a '#C' /
bind '#|' /tmp/pipes
cat /tmp/pipes/data > /dev/hoststdout&
cat /dev/hoststdin > /tmp/pipes/data&
export / <>/tmp/pipes/data1 >[2] /dev/null
echo halt > /dev/sysctl

The problem is that if I terminate the local inferno, the remote emu still runs.  I'm not sure how to catch and clean that up gracefully....

converting push to pull

2009-05-27T10:16:00.003-05:00

Forsyth pointed me at the Owen papers with the labor exchange methodology for grid workload management. It works more on a pull principal (as does Cilk workload-stealing come to think of it).

The basic model behind PUSH's distributed pipelines are that the workers "pull" their data from the pipes -- but the problem is that the work is currently uniformly distribtued by the fan-out process. What we need is the equivilent of an alt which will distribute new work to workers with the capacity for it versus getting jammed out round-robin'ing work.

Call for Interns

2009-05-26T08:57:00.002-05:00

I'm starting the selection process for a 6 month intern position working on Plan 9 technology for the DOE project (details at http://www.research.ibm.com/hare). The 6 months would have to be between Sept of this year and August of next. Graduate students or post graduates preferred. You'll need a strong background in C programming with Plan 9 and/or Inferno experience preferred. If interested, I need a resume and availability by this time next week (June 2, 2009) and we'll need to chat on the phone by the end of next week.

There will be a second opportunity for a 3 month position for a separate (but still somewhat related) project - I'll post more details on that project when it becomes available -- it may be during next summer (2010).

Please send resumes, recommendation letters, and availability to me (ericvh AT gmail DOT com).

lcmd games and 9P tunneling over ssh

2009-05-25T17:03:00.003-05:00

So I've started playing around with lcmd (see ipn lab 83 for more details) -- and its got the right data path for what I want, but I need a slightly different application than simple rcmd.

Effectively I want to use os from one Inferno to ssh to another machine and then use the ssh pipe to allow the two Inferno instances to mount eachother. The will allow me to securely traverse an ssh-only firewall and get more or less full service connectivity between the two inferno's (so that one can use the others network, or file system, or underlying host, etc.)

First stage is to use the ssh tunnel to either import or export a file system - I suppose import makes the most sense initially. The second stage is to build a 9P mux which differentiates which pipe to service depending on the 9P message header (directing responses to a client pipe and requests to a server pipe).

Simple experiments with lcmd seem to fail under OSX -- but I was trying to do them between acme-sac and emu, so its possible that one or both have issues with their hoststdio implementation (of course it could be something else I'm doing wrong as well) -- I do notice slight differences in behavior with hoststdin between the two -- acme-sac seems to report all activity, while regular emu seems to be muxing between a process reading hoststdin and another thread (perhaps the shell).

Come to think of it, the ipn stuff does an enormous amount of work to mux the hoststdin and hoststdout into a single pipe, why not just do this from the get-go?  This applies to both hoststdio as well as the os command stdio -- easy enough to fix in both cases (I think).

In the case of the OS command, its just in how its written.  The underlying devcmd just differentiates at the op level, not at the open level.  So we just need to modify a line in os.b to use the same descriptor for both stdin and stdout. The hostio isn't quite as straightforward due to some hard-coded permissions.  In order to preserve existing semantics, i'm just going to create a new Qid for r/w io.

The other problem here is that the target system is actually linux-powerpc. An open question is whether to port acme-sac there just to keep the source base consistent between client and server, or whether to attempt to use traditional emu as the server (implying changes in both).

acme sac does seem much better under OSX

2009-05-22T13:06:00.003-05:00

Wow - I guess I had never really tried ACME-sac out on OSX. It seems much better integrated.  The mouse scrolling works (which doesn't sound like much of anything, but really makes a difference), the default fonts better match the Mac. Its making me reconsider some of the client-side configurations I was considering for a new working environment for dealing with Blue Gene.

So -- what would be the implications of going with ACME-sac for the front-end instead of a vanilla Inferno? There was some consideration of using Inferno as a drawterm replacement, but that sort of faded due to latency concerns. 9cpu seems to be present, but broken -- perhaps its something that is easily fixed.

EDIT: Of course trying to fix things ends up revealing how much more restricted of an environment it is. I was never a huge fan of the Inferno wm-based debugger, but it did make problem determination significantly easier than without it. The case could easily be made that the right thing to do is "port" some of these wm applications to acme -- the common difficulty is one of time to do these sort of things.

So, what's the more pain in the ass, back-porting improvements from acme-sac into inferno or porting over "missing" items to acme-sac? "If I were king", I suppose I would take a slightly different approach:
# kill the binaries in both targets to ease source control woes mentioned earlier
# structure a generic build script like p9p to allow easy builds without binaries and modifying mkconfig every, freaking, time
# create a distribution target that would build the acme-sac configuration directly out of inferno

Of course, I'm not in direct control of either of these distributions, but I think if I do branch Inferno for Brasil, maybe I'll follow this guidance myself.

Source Control Woes

2009-05-22T10:32:00.002-05:00

Blech. I've been trying to work in a branch of my mirror of the inferno-os tree on google code. The problem is that it contains a bunch of binaries which royally screws with the source control system.  I removed most of the binaries, but of course now I can't pull from that tree without it freaking out.

As far as I can see, I have two alternatives: 

* meticulously create (and maintain) a gitignore file so that every time I rebuild it won't put binary changes into the commit. This will (probably) fix the merge issues, but I'm not certain.

* create a full fork of the inferno-os tree and manually patch source files based on the changesets from svn.  Actually, it would probably make the most sense to just fork a brazil-os tree from the inferno-os tree and strip out absolutely everything we don't really need.

EDIT: I've gone with gitignore for now, but a bit unsure how to commit the file back to the central repository.

UEM: Incremental Approach

2009-05-20T16:16:00.002-05:00

I realized I was getting a bit overwhelmed by the complexity of what I described in the whitepaper and it was leading me to write more and implement less. I'm going to switch gears and go for some small, incremental steps to get the ball rolling.

First things first, I need a method for initiating commands on Inferno hosts using only a synthetic file system. Noah has the basics for this within exportcmd, but I need to go over it and make sure it fits my goals.

Second, I need the ability to mount that file server on OSX, Linux, and Plan 9 -- preferably over the stdio of the Inferno
process. There's an Inferno Lab which covers something similar that I can use as a base.

Next, I'll need to be able remotely mount this service while simultaneously exporting my namespace to it by using a mux on the communication channel which appropriately directs requests.

Then I can get into extensions which are necessary for devcmd, but the more I think about it, the more I think many of these extensions may be better implemented from an application level versus trying to do them within the driver (mostly things to do with namespace manipulation).

Also (where possible), I'll need a mechanism within devcmd to actually back mount an Inferno sandbox where I can setup the namespace for the target application.

Once all of this is in place I can move on to thinking about the front-end file system which is used to reserve resources and allocate nodes.  I'm leaving this for last because it will require a plugin module and I'm still not entirely sure what form that will take.

BG/P

2009-05-20T13:57:00.003-05:00

The other approach I could take would be finally digging in and reworking my environment for Blue Gene using elements of the UEM. There is more complexity to a complete solution here, but I could start with the basics and get rid of the Plan 9/Linux/cobalt/ssh scripts I use now, unifying into a single consistent environment which I can completely host from my laptop instead of having to involve intermediate CPU servers. The advantage of doing this will also be to ease others who need to transition to using BG/P.

There are two sides to the BG/P story -- there's initial machine allocation and then there is an environment for actually executing applications on the allocated machine.

For these there is less emphasis on cloud-like allocation like with Kirin, but more of a focus on the BG/P boot environment combined with clever network tricks to traverse the multiple network domains involved.

UEM Pre-Implementation Notes

2009-05-20T09:53:00.004-05:00

I need to lay out a pragmatic model for getting the execution model prototype going. In order to keep one of my interested end-users I'm going to need to rapidly prototype a version which fits his current needs and then build upon that in order to obtain the more general model. Leaving out the loftier goals for zero-configuration and plug-in modularity, I think the easiest approach would be to augment devcmd with sufficient additional files and control interfaces to provide a more complete xcpu set of facilities and information (without treespawn). We can then build a front-end task-based control interface using exportcmd as a foundation which interacts with the back-end augmented command facilities.

Since Inferno will be the baseline for these activities, I guess it makes the most sense to operate under a full fork (at least for now) -- I have had thoughts of stripping an Inferno branch down to a bare minimal set to meet my purposes, but for now I think it just makes sense to use a proper fork in case I change my mind and forsyth wants to accept changes mainline. Oops - I guess things don't really work that way, since I own the inferno-os repository I can't do a proper repo fork, I'll need to operate within a branch for now (which is probably more efficient space wise anyways.

Also, one /proc we didn't include in our description was the Linux proc which has grown quite signifigantly (I guess OSX doesn't have a proc? - I thought BSD did). Gonna update white paper with interesting tidbit of linux proc leafs. I wonder how many of these we should give access to (if not all, I suppose that would make the most sense but would change the character of devcmd somewhat - most specifically
in directly linking it to an underlying process id. That's an interesting concept anyways I suppose - of course different data would be available on different host operating systems (Plan 9, Linux, and OSX being the interesting portions in our case. In the OSX case there won't be a procfs to access the data through (unless we use the FUSE version, but it would seem we would be better off directly providing the interfaces within Inferno instead of using a proxy).

A related idea would be to provide file system interfaces which could be used to attach gdb to these remote tasks. Since GDB doesn't really work on synthetic file systems like acid, I suppose the best approach would be to just provide a ctl message which sets up a remote gdb slave attached to the process in question. Is that sufficient to allow remote debugging, or do we need something more? Of course, in a cluster we really want a meta-gdb capable of controlling multiple tasks on multiple systems. ACID really does seem to have a leg-up here.

cmd should have sufficient infrastructure for the backend (at least for running host threads which is the first thing we are concerned with, we can accommodate running native Inferno threads later). The front-end may be a bit more challenging, based on the example write-ups we did it seems clear to me that the task based model is likely the most natural for Damir's programing model -- but we'll want top-level reservation control, so that implies a slightly deeper hierarchy than the existing exportcmd. As such, we'll need a top level clone directory which specifies tasks, the tasks directory will have their own set of files which will allow aggregate command execution or data collection. Then we'll have a set of subdirectories for each thread of execution with specific data and control files for interacting with the thread (which will ultimately be a binding of the backend devcmd session directory).

The difficult part for GPUs is distributing those threads based on available resources (and doing it in such a way that the technique is generalizable). My current thoughts on this is to use a generic key/value tagging facility to specify what resources
are available on each system and use ctl messages to (atomically?) manipulate use-count flags. This needs to be thought through a bit more. The other difficulty here is hooking into the GPU tools (in this case Nvidia runtimes) to setup what resources are available. Fortunately (from an implementation perspective anyways), the lack of external interfaces for querying the GPUs means there isn't a whole lot of information for me to publish or update.

There's an interesting side issue here which is who is the agent that collects and updates the information. For the system itself (load average, available memory, number of cores, architecture), it may make sense to do it within the architecture specific cmd implementation. But for something like GPUs (which may or may not be on a particular architecture) - it may make more sense to use a limbo plugin which uses the native devcmd to query the device and update attributes and tags. The question is to know when to run these helpers. For the pragmatic prototype, we'll always run them since we'll always have a GPU present on the system -- but we'll need a long range plan versus always running every potential back-end plugin to query the device. I suppose something which scans (and rescans for hotplug?) the host peripherals may be necessary. Suggestions welcome on this.

A Few Words on Brasil

2009-05-20T09:42:00.004-05:00

The astute will notice the brasil tag at the bottom of many of these posts - since its genesis predates the use of this blog for brain dumping, I think its appropriate to briefly describe what "brasil" is.

Brasil is intended to be a meta-project for a collection of experiments relating to non-incremental changes to the Plan 9 operating system. At the moment, most of these ideas are focused around the needs of the DOE HARE project, but this may not always be the case. The main experiments currently in the roadmap are as follows:

* Create a Plan 9 microkernel where the kernel only provides the IPC mux for 9P messages, this will include the migration of all non-synthetic-file-system based system services (such as the scheduler) to file systems.
* A new unified registry model with multiple scopes facilitating dynamic distributed resource access without static configuration files or reliance on DNS
* A Unified Execution Model which extends across the network and contains support for creation and management of logical node instances and physical hardware as well as threads.
* A peer-to-peer DHT approach to storage and file systems including transitive mount short-circuiting support, caches, distributed lock management.
* Semantic file systems which have more of the properties of databases versus flat file trees.
* Additional library support for message passing style interprocess communication and synchronization.
* Synthetic file systems incorporating collective communications semantics
* web-browser targeted synthetic file system user interface model with tightly bound collaboration components (fluff)

I'm using modifications of hosted Inferno as a prototyping platform for everything except for the microkernel work.  The concept being that I can implement infrastructure in Inferno and use it natively on Plan 9, Linux, and OSX (among others). More specifically, I plan to take a co-operating-system approach where Inferno exports all of the interfaces to these services as synthetic file systems to the host (which use 9P, v9fs, or MacFUSE to mount the resources within the system's namespace).

I have speculated about creating a stripped down version of Inferno (without graphics, the virtual machine, or other components which I don't currently need) to keep things as tight as possible, but for now am doing my work within a full Inferno tree.

Not that it matters much, but the name itself is stolen from the "code name" (perhaps that's not the right word) for an evolution of the Plan 9 system in the early 90's which was renamed Plan 9 in the late 90's (cite).

Alternatively it can be an acronym:
Basic Resource Aggregate System Interface Layer
(or perhaps more descriptive of the prototype)
Basic Resource Aggregate System Inferno Layer

Next Steps: UEM

2009-05-20T09:29:00.003-05:00

Based on early feedback, there are several points that require additional clarification within the unified execution model (UEM) design description.  I need to spend some time writing up sections (which will manifest themsleves as entries within the blog) on the following topics.

* More detail on semantic file systems, including descriptions of how they can be used for search and monitoring as well as specifying attribute requirements for provisioning.
* Provide detail on the underlying design and implementation model for user and system defined tagging of objects with key/value pairs.
* Need to start defining the core semantic and syntax specification for back-end elements as well as core models for certain classes of back-end resources in order to optimize re-use of tools which may use the file system
* Need to further clarify best-effort approach for guessing what resources a user/application wants based on existing context
* Need to identify a motivating example to frame/focus LADIS paper around.
* Need to write up multi-dimensional file views in a more consistent factor to get the idea across to folks that aren't as familiar, and I need to include a more formal write-up on the dot-dot-dot shortcut idea.
* Need to describe the canonical form reference fs dimension and describe how it can be used to short-circuit transitive mounts and otherwise be used as a sort of global file reference. Go on to compare/contrast this to the idea of a global file descriptor.

Unified Execution Model

2009-05-19T19:46:00.011-05:00

This is a mind to whitepaper brain dump I've been working through with Noah and Pip -- its rough, incomplete, and may not make a lot of sense.

Motivation

While Plan 9 established a cohesive model for accessing distributed resources across a cluster, it employed only a rudimentary methodology for initiating and controlling remote execution. More specifically, the cpu(1) facility provided a mechanism to initiate remote execution while providing transparent access to certain aspects of the initiating terminal's resources using Plan 9's dynamic private namespace facilities. While the cpu(1) facility provided an elegant mechanism for remote execution, it was limited to a single remote node, which was explicitly selected either by the user or DNS configuration. This worked well enough for small clusters of terminals with a few large scale-up CPU servers, but is less appropriate for today's scale-out clouds.

Breaking from Plan 9's paradigm of representing everything as a synthetic file system, the cpu(1) facility was instead provided as a more traditional network service, giving it less flexibility than many of the other services which were represented in the file system and could be reorganized in arbitrary ways or exported over heterogenous remote networks. This also restricted access to remote execution to systems which were directly connected to the same network as the initiating terminal, in other words firewalls and separate network domains created additional hurdles to initiating remote execution.

As we deploy Plan 9 to larger scale clusters with heterogeneous network configurations, there has been an increasing need to address the limitations in the current remote execution model. In my personal experience, three classes of cluster configuration require something beyond the current mechanisms: clouds, hybrid systems, and extreme scale HPC installations.

Cloud

Clouds introduce a signifigant new factor to cluster configurations in that they facilitate a fluid allocation, provision, and configuration of services. As end-users require new resources, they can merely provision them, dynamically hooking them into virtualized storage or virtualized networks (which themselves are organized dynamically). Additionally, clouds themselves are typically composed of a large number of smaller nodes versus a few large SMPs.

Services such as Amazon's EC2 allow end-users to dynamically provision new resources programmatically, with the new resources brought online in the time span of minutes versus the hours (or days) it would typically take to order, build and configure a server. Services may be deprovisioned at a similar pace, allowing users to scale back the expense of hosting a service when demand is low.

This fluid environment with resources appearing and disappearing at regular intervals, static configuration is no longer an acceptable strategy. Host configuration either using Zeroconf or some other form of registry will have to be the norm in order to reach the potential of the dynamic cloud. The flexibility from an administrative standpoint is only half the cloud story, as applications are enabled to demand and release cluster resources we have the potential for entering into a new golden age of distributes systems with Amazon, Google, and other service providers providing the appearance of limitless resources for distributed computation.

Hybrid

Figure 1: Hybrid Topologies

Another recent trend is the rise of hybrid system and hybrid cluster models. Hybrid systems incorporate heterogeneous cores, either with different processing characteristics or difference instruction set architectures. The Cell Processor, with its combination of a general purpose PowerPC cores (PPU) and eight synergistic processing elements (SPE) for vector operations is a classic example. In such systems, the various cores have coherent access to eachother's memory, but memory transfers between PPU and SPE are handled explicitly.

A slightly more loosely coupled version of this can be seen in systems using GPUs to accelerate compute. In such systems, the GPU accelerators are often located on the other side of a peripheral bus with their own memory and execution units. Existing runtime frameworks for GPUs, such as CUDA, provide preciously little introspection and control to the external system - making monitoring and control difficult.

The Road Runner tri-blades used a set of cell processors connected over peripheral buses to a general purpose opteron combining both of the previous examples. Road Runner poses many interesting challenges as a Hybrid as it has multiple instruction sets, multiple endian, and multiple peripheral buses.

These sorts of hybrid systems can themselves be composed together with other more general purpose systems in a hybrid clusters. The Virtual PowerXcell Environment (VPE) was an attempt to create a runtime which addressed such a loosely coupled hybrid cluster by trying to maintain the illusion of a single system image.

Extreme Scale

A final area of particular interest is extreme scale computing. Blue Gene is capable of having hundreds of thousands of cores, with subsequent versions aiming for millions. Managing and debugging the physical hardware at this scale presents a challenge, deploying, monitoring and controlling application threads if even more daunting. The lack of infrastructure capable of coping with this extreme scale, has lead to oversimplified application frameworks only capable of statically defined and managed embarassingly parallel applications. We believe the development of a dynamic command and control framework for extreme scale is the key to unlocking its true potential and broadening its applications base.

Beyond sheer scale, systems such as Blue Gene present a number of interesting challenges for execution frameworks. The machine employs a physical paritioning mechanism to allow it to be used concurrently by multiple users. The physical partition scheme is limited by the underlying machine architecture, creating a minimum allocation of one I/O node and 64 compute nodes (at least at Argonne, smaller granularities are possible, but the smallest is one I/O node to 8 compute nodes). As a result, when requesting additional resources, you actually are requesting a private cluster composed of some number of I/O nodes (which are good for running certain types of tasks) and a certain number of compute nodes (which are good for running another type of tasks). This complicates the cloud scenarios discussed earlier.

Figure 2: Extreme Scale Topologies

Furthermore, extreme scale systems such as Blue Gene incorporate several underlying physical network technologies which creates several different network domains which cluster elements have to work within. For example, end-user workstations can only access the front-end nodes (which are general purpose PowerPC servers). The front-end servers can talk directly to the management infrastructure as well as the IO Nodes, but not the compute nodes. The Compute Nodes can talk to the IO nodes over the tree network, but not the high-bandwidth torus network. Coordinating management components (not to mention applications) in such a diverse topology is challenging.

Requirements

While primarily intended as a research project, it is important that whatever solution is developed is deployable in a number of contexts including Linux, MacOSX, and Plan 9. It needs to be able to support Intel and PowerPC architectures (in 32 bit and 64 bit varietys) and it is desirable that it incorporate support for accelerators such as GPUs and SPUs. It needs to be network agnostic, requiring only a reliable, in-order tansport - and it needs to support some forms of dynamic configuration and shouldn't require static configuration files to work.

In order to accomodate cloud configurations, an execution model must not only be able to initiate remote applications, but potentially to provision new nodes to initiate remote applications on. This essentially also implies control of booting (and shutting down) system instances. On clouds which support it, it might also incorporate controls for migrating system images (and contained applications) to new physical hardware.

Resource management software, such as load balancers will have to monitor the currently allocated assets and know when to acquire more or release existing resources in order to adapt to changing workloads. Since the cloud environments have multiple network security domains, such an execution model will require mechanisms to be able to cope with traversing the multiple network spaces to maintain monitoring and control of remote processes.

With so many different resources spread out across the cluster, users and applications will need new tools to be able to keep track of what resources they are currently being charged for and how they are being used.

Additionally, there are cases were it is desirable to run many instances of an application on many remote nodes, breaking the original one-to-one model of cpu(1). We've seen this requirement take two forms: in one form the important thing is just to start many duplicate copies of the application with the same arguments and environment. In such cases, it is the applications responsibility to configure each instance appropriately. Standard I/O in these cases is primarily a logging mechanism as opposed to a mechanism for interaction or reporting data. The other case requires the ability to spawn off many instances, with individual configurations and control channels. Finally, while in most cases the applications are spawned from a single terminal or front-end node - we feel it is important that new instances (or resources) may be allocated from within the cluster as well as from the front-end.

For heterogeneous systems, we must have the ability to provision systems with specific characteristics (ISA, Accelerator Properties, Memory, etc.) on specific nodes. We must also be able to accomodate infrastructures such as CUDA which combine execution on the general-purpose CPU and the GPU accelerators. As such, execution must be conditionalized on being able to find sufficient resources to initiate execution. In environments which support it, such as Cell, the execution model should provide controls to hybrid accelerator hardware.

Since hybrid models such as CUDA operate as black boxes, we must be able to support dedicating hardware to a particular task as well as standard time sharing models - incorporating the same basic features as a cluster provisioning system such as cobalt or the sun-grid-engine at a system image and/or task level.

Given that hybrid accelerators may be present over tightly coupled peripheral buses (or even on chip) - whever mechanisms are used to establish command/control must be transport independent (ie. able to run over a peripheral bus as well as over a standard IP network).

In order to be able to use the execution model at systems with the scale of Blue Gene (or potentially larger), the execution model interface needs an organizational interface which can scale to millions of threads. To be complete, it will have to handle physical resource provisioning at multiple granularities (to match underlying cluster management tools) and provide mechanisms to access hardware debug facilities.

Application threads at extreme scale often require extremely low latency access to computational peers, adding the requirements of data locality and physical topological awareness to any workload distribution and management system. Furthermore, it is likely that such applications will require similar knowledge be integrated into whatever inter-thread communication and synchronization primitives provided by the framework.

Application (and system image) deployment at large scale also provides a challenging problem. Some manner of aggregated deployment of application images (and data) will be necessary in order to accomodate simulataneous execution of thousands of new threads.

Furthermore, the overhead of monitoring such a large scale system is signifigant. Any monitoring/reporting infrastructure will have to be augmented so that the overheads of monitoring nodes does not overwhelm their communication or computation capability.

Finally, the sheer number of components of such large scale systems make failure of individual components more likely. The systems infrastructure for providing the execution model should be resilient in the face of such errors, migrating workloads as appropriate and not failing itself when it can no longer contact some of its components.

Related Work

Suffice to say, there have been a number of projects attempting to address distributed execution models. Within the space of the motivating examples, the two most prominent paradigms are Map/Reduce and MPI mechanisms - both of which were designed with a particular application structure in mind. We seek a more general-purpose execution model capable of handling the aforementioned requirements.

The XCPU runtime infrastructure was an attempt at bringing the power of Plan 9's cpu(1) facility to HPC systems running other operating systems. It incorporated facilities for dealing starting large numbers of threads on large numbers of nodes. It optimizes binary deployment using the treespawn mechanism which allows it to task clients as servers to aggregate the work. Its associated monitoring infrastructure uses s-expressions to more easily allow collection of correlated remote monitoring data from nodes. The xcpu client also had provisions for supporting multi-architecture hosts.

XCPU suffers from a rather ad-hoc authentication mechanism which isn't persistent and ends up being a bit of a pain to deal with. It doesn't incorporate workload scheduling itself, relying instead on external tools. It also suffers from static configuration, and doesn't easily support fanning out with individual arguments and I/O paths. Finally, XCPU's currently implementation is linked to scoket transports, and while it supports the use of private namespaces and limited binding in XCPU2, it only does so on modern Linux kernels.

There are a large number of process style interfaces both to local processes and remote clusters. These are summarized here for reference:

Design

The core idea is to build upon the ideas present in xcpu, xcpu2, and to some extent spufs, cellfs and kvmfs to provide a comprehensive execution model which covers allocation and management of logical nodes as well as distributed execution and to some extent communication. The idea is that this could provide a more dynamic execution model for EC2 deployments of Plan 9 and Inferno as well as provide the foundation for large scale deployments of applications as part of the DOE HARE project. This should also fill the distributed execution requirements for the PUSH project (which in turn will provide facilities for connecting distributed execution in a pipeline fashion).

The core interface principle remains as a service-oriented file system based on the same basic design as proc(3) or Inferno's prog(3). The idea, is at its core, each thread will have its own directory with a number of control files.

The basic design builds upon these past designs by using clone files to allocate new resources in much the same fashion as the cmd device from Inferno or xcpufs. Each newly allocated resource will have a subdirectory containing control/status files specific to the resource which the directory represents. Certain standard files are provided at every level (for instance ctl and status) - but the semantics/syntax of interacting with them may be more specific to the underlying resource.

The use of clone files to instantiate threads may be a bit controversial as it always comes up as the example of how we don't want to use a file system for everything. However, I believe the need for distributed execution contexts trumps prior reasons for not instantiating new threads through a file system interface. Its not clear that there should be a complete switchover to using file system operations for things such as fork, but it seems clear that we need an architected mechanism for execution in other contexts. By using a synthetic file system for this interface, it means we can easily export it to other environments - all the system would need is a 9P client library or file system in order to access the execution model.

For example - Inferno and Plan 9 have different files based on what underlying information and controls are avaialble. Given the wide space we intend to address, this will likely be the case for us as well.

The sheer number of nodes and threads we intend to support will not be possible with a flat organizational structure. One approach would be to use a hierarchical structure matching the physical organization of the nodes (ie. data center -> row -> rack -> chassis -> slot -> lpar -> thread for a blade configuration). Such a model might have multiple clone files at multiple levels providing abilities to intantiate new logical nodes, logical partitions, or threads. Another model would be to match the logical organization of a cluster management tool (i.e. job number -> IO Node -> CPU Node -> Core or Thread for Blue Gene). A similar model would be possible for Cloud configurations. Another related model would be to use network topology in order to address resources. This would provide semantic information in order to directly contact or allocate neighbors in a torus or other locality based topology.

Perhaps the most flexible approach would be to allow users to define an arbitrary hierarchical organizational structure using a /n-like dynamically generated organization terminated by a clone file to allocate resources at that level of the hierarchy. In cloud configurations, such a dynamic hierarchy could also be used to help allocate machines with specific attributes (i.e. /cloud/x86/gpus=1/mem=4G/clone). The same technique could be a general mechanism to find hardware capable of supporting the objtype of a given application (or allocating the hardware as necessary). There should be a relative mechanism available which chooses reasonable defaults based on the currently executing application such that a user doesn't have to specify an exhaustive list of attributes for every innvocation.

The likely case is that any chosen organizational structure will not be optimal for every type of access. That is to say, one organization will not be sufficient. As such, it is desirable to provide multiple views into the model: supporting physical, logical, and user-defined views. This should be possible through the use of a generic key/value tagging facility which should provide sufficient flexibility to satisfy most constraints. This same solution should be able to work with both physical resources as well as user-defined tasks and logical resources.

Another parameter which might be useful in such organizations is to provide scoped dynamic attribute specified access. That is to say, provide a subdirectory which when traversed will perform resource lookup based on the scope of the current level of the hierarchy. This could be used to find a node within the current logical group which has a certain resource available, or be used to provide flat access to all processes within the current logical group.

Finding a mechanism to be able to switch between these alternate views of the unified execution model will be critical. A shortcut mechanism such as dot-dot-dot may be appropriate here.

Due to the variety of underlying resources, it is desirable to provide a plug-in interface for both the organizational structure of the execution model as well as the thread-level elements for different back-ends. This will allow for new resources to be added to the model with a minimum of fuss, and allow for easy construction of unforseen organizational structures.

An vital design element is that the unified execution model is not a centralized resource. Each element of the system which can will run their own instance of the synethetic service-oriented file system. These instances will coordinate with one another in a fashion appropriate to their underlying network topology. Since many nodes will never have to contact one another, cross-mounts are lazy and initiated on access.

This approach implies that every client is also a server. Since we want to be able to traverse network boundries such as firewalls, NAT appliances, and bridges between different network technologies it is important that we are able to use a single connection for the two nodes to be able to access eachothers resources and provide mechanisms for nodes to be able to transitively access eachother through bridge nodes. In the future, these relationships will be managed by retooling the srv(3) mechanisms to be more cluster aware - but in the short term we can experiment with these different methodologies directly within the execution model.

Aggregated data access is a critical element of the execution model in order to be able to cope with scale. It will be unacceptable to issue millions of directory traversals and individual file operations in order to obtain a snapshot of the cluster's state. As such, directory data must be available in a compressed alternate form such as the s-expressions employed by xcpufs.

Implementation

Hosted Inferno seems a natural choice to implement these designs. It already has its own authentication mechanism and full dynamic private namespace facilities. It runs on all target operating systems and provides a nice abstraction of their underlying differences. Furthermore, through the cmd(3) infrastructure it is able to initiate and control processes on the underlying system. An interesting aspect of this setup is that we can start both Inferno threads or host threads using roughly the same interface - that can be extended to initiation of CUDA threads and/or emulated environments (such as linuxemu on Plan 9 or cnkemu on BG/P Plan 9). These threads can access resources such as the initiating nodes file system through a 9P backmount which can be organized and exported via Inferno (MacOSX will have to use FUSE to do the back-mount, but this allows us an Inferno pseudo-dynamic-private namespace on OSX).

Additionally, the general idea is for Inferno to export the synthetic file system interface of the exection model to the underlying host allowing it to directly access the interfaces and initiate actions either via shell scripts of library APIs. This is why it is of vital important that all interaction should be possible using only the file system based interface.

A potential gap is that Inferno does not currently support .U extensions necessary for some Linux applications. We'll need to look into mechanisms for either implementing .U semantics within Inferno or bridging them to remote servers which implement them.

While Inferno may seem a bit heavy weight for this task, consider that the text size of many comparable infrastructures is far greater. Even XCPU isn't that much smaller than a full blown Inferno execution. In the future this overhead may be further diminished by creating an Inferno stand-alone-complex for the execution model which eliminates the virtual machine and graphical components of the hosted inferno emulator. The working term I've been using for such a stripped environment is the Basic Resource Aggregation System Inferno Layer (aka Brasil).

Limbo's dynamic module loading seems to be a fairly optimal solution for the integration of plugins. A file based interface (perhaps even a plumbed one) can be used to instruct the execution model to load new plugins dynamically - relying on a static configuration to load the defaults for a particular system (and/or user).

It seems clear that there are at least two different forms of plugins: back-ends which supports allocation, execution, monitoring, and control of specific physical or logical resources (clusters, nodes, logical partitions, threads, accelerators, etc.) and front-ends which support different organizational models (physical hierarchy, logical hierarchy, attribute based search, task-based, etc.). The existence of different underlying transport mechanisms may require plugin support as well - but it is less clear to me how that can be easily integrated and may be best left to a separate infrastructure (such as /net or /srv).

Back-end plugins will provide two levels of directories - a toplevel which is the same for each physical instance of the resource group, and an instance directory with the per-instance control and status interfaces. These two level directories can be stacked (such that per-instance directories might contain another resource's toplevel and instance subdirectories). This allows compositional hierarchies to be easily built. TODO: figure out syntax of how this is going to play out via the file system or modular interface.

Access to compressed aggregated information about a subset of the resources will be made available through an alternate attach point of the synthetic file system (and optionally through a dot-dot-dot shortcut).

Communication and coordination of different instances of the execution model will be accomplished via a single pipe. Client and Server messages will be sent on the same pipe, with a special mux on either end capable of distributing requests or responses as appropriate. This will allow communication between two hosts when only one of the hosts is addressable, it will also allow easier tunneling of communication paths. Extensions on this mux to allow bridging of network domains will also have to be investigated.

The essential idea is that each participant in the unified environment will run their own instance of the execution model. They will communicate directly with that instance in order to interact with other nodes (through their instances). In other words, every thread talks to its local instance, and the instances are responsible for coordinating with eachother. This keeps the model decentralized and should allow for enhanced latencies when dealing with local resources versus remote resources.

Examples

In order to validate the interfaces described above, I'd like to run through a few scenarios of user or application interaction with Brasil.

Kirin Hybrid Cluster

The first use case we will explore is the use of this mechanism within our KIRIN cluster, which is a set of opteron workstations running Linux with GPU peripherals and CUDA runtimes. We want to be able to allocate machines (and GPUs) to run distributed applications on, and have some rudimentary communication mechanisms for control of the threads of the application as well as a communications channel for communicating results back to the initiating host as segments of computation complete. In our case, the initiating host is a Mac Pro laptop running its own instance of CUDA in order to visualize the results of the computation.

For the purposes of this example, lets say our application is a computational intensive imaging application. The image to be processed is a large file on the disk of the initiating host. The front-end application is setup to split up the image into uniform tiles and provide the tiles to the distributed worker threads to process and return the processed tile back to the initiating host.

The master thread parses the image file to a sufficent degree in order to determine an upper bound on the number of logical instances which would be required to process the image completely in parallel. The user, either by parameter, configuration file, or environment may limit the maximum number of requested logical gpu instances. It then requests reservation of that many GPU instances across the cluster and initiates execution of subject threads built using the CUDA runtimes. These subject threads then block on reading their standard-in to request work. The master thread, who has references to the stdin, stdout, and stderr of each subject thread instance issues work assignments by sending a relative path to the master file along with coordinates specifying the tile that the thread in question is instructed to process. It is important to note here that, following the xcpu2 model, the subject threads have the same file system namespace as the initiating thread. As threads complete their assigned work, they can either write an image file of their own to the shared file system or send the results over standard out directly to the master thread. Upon receiving results, the master thread can issue more work, or instruct the subject thread to terminate and release the resource.

Figure 3: Kirin Task Example

The master thread can coordinate this activity and communication directly using the synthetic file system, but it is much more likely that it will do so via a simple library API. Specialized versions of this library can be constructed for certain classes of application and provide more advanced facilities such as Cilk-like work stealing, data pipelining between computational elements (ala PUSH), or transactional semantics enabling either checkpoint/restart or triple modular redundancy of computation. Eventually, it would be nice to incorporate these facilities into runtimes which manage issues involving different instruction set architectures.

From an implementation perspective, we really have two different organizational elements: physical resource allocation and task based execution. The first thing we need to do is allocate some number of GPU nodes where the maximum is defined by user specification or the actual available resources. There is not really an existing example mechanism which allows us to allocate a group of physical resources without logical sharing or blocking. We could query existing available resources and then dispatch tasks using the clone mechanism - but there is an inherent race condition between the checking of the resources and the dispatch of children.

A strawman approach is to use a task-based organization - the master thread opens the clone file on the task view of the execution model. This clone file creates a new task session id, and creates a subdirectory with that session id. Subsequent reads and writes to the fd which was opened as the clone file will be directed to the ctl file of that subdirectory following the typical clone semantic of Plan 9.

For the purposes of this strawman, lets say his image has 100 tiles to process. He issues a command to reserve 100 GPUs, and gets an error response that only 4 GPUs are available. He re-issues the request asking for 8 GPUs, which succeeds. This creates 8 new subdirectories in the task session directory, each representing a thread on the local or remote system which has been allocated a GPU resource. Since he is executing the same subject thread on the resources he can use his existing fd handle to issue the execute command to all subject threads. The master thread can then open the data file of each subdirectory in order to obtain a communications handle to the remote threads on which he can issue work assignments and read results (or pointers to results).

When finished, he can cleanup the allocated resources by closing the handle to the ctl file. Individual threads can be shutdown or retasked by opening their individual ctl files present in the subject thread subdirectories.

BG/P Allocation and Debug

On more of an infrastructure focus, I'd like to use the unified execution model to be able to allocate resources and provide hooks to debug the kernel using either JTAG hardware (for BG/P) or hooks in the hypervisor (for Cloud). As mentioned earlier, Blue Gene presents an interesting problem in that physical partitions of the machine have to be allocated at some fixed granularity (such as 1 IO node and 64 compute nodes on the BG/P at ANL). Another interesting factor is that physical machine allocations are handled by a batch-job allocator underneath. So, we need to be able to cope with group allocations of multiple granularities, and must be prepared for waiting for extended period of times while the parition is allocated or resources for that size partition become available.

A model similar to the resource reservation methods described in the Kirin cluster should allow us to request various physical partition sizes and characteristics. Instead of failing when resources aren't available, an option to the reservation can choose to block the request until resources become available. Since Blue Gene can be configured to boot different kernel images, the reservation mechanism can specify alternate kernel profiles and/or parameterization to the various kernels. Configuration commands needs to be set prior to issuing the reservation command.

The top level control file can be used to issue commands to Cobalt (such as terminating an allocation) and can also be used to query both the status of our task and obtaining the status of other runs on the machine and direct access to various machine log files.

The hierarchical nature of Blue Gene's physical topology creates another interesting difference. When the reservation successfully completes, the top level session directories will be for the IO nodes for a particular logical partition. The compute nodes accessible via that IO node will be represented in session directories under the IO node's session directory.

For Blue Gene we are interested in both remote command execution and physical node access and debug. As such, in addition to the normal execution control and status files in the IO node and Compute Node directories, we'll also provide top-level debug and memory access. Since each of the cores on a BG/P have separate hardware registers, we'll provide access to the per-core registers in subdirectories. Since the normal session syntax uses numbers which will relate to processes executing on that node, we'll use alphabetically identifiers (A, B, C, D) for the various cores. Debug commands issued to the ctl file in the core subdirectories will only effect that core, while debug commands issued at the top level will effect the entire node.

While initiating tasks from the physical hierarchy view can be useful, it is much more likely we'll want other configurations for starting new tasks. One possibility is the desire to start a task on a neighbor node in the torus. In order to do this, we'll need to provide an alternate organizational topology for the node session access which takes the form of the Torus's three dimensional configuration. This can provided in an alternate attach namespace reachable from the partition's top-level or through a dot-dot-dot shortcut. Beyond physical or network topology, it may be desirable to provide user allocated node groupings, this will be accomplished through a new special session control file named tag which will be present at all levels of the hierarchy and can be used to apply new tags to individual nodes or nodes and their children. Access to nodes using these tags can be done through a semantic search hierarchy also accessible from the top level of the partition session directory.

So, to execute a Monte Carlo simulation on a BG/P, the Master Thread of the application needs to first request a BG/P partition of the appropriate size (lets say 512 nodes). If that size allocation is not immediately available, it returns an error, and the master thread re-requests the 512 nodes with an option which blocks the read until the resources become available. Once the resources become available, the read unblocks.

The user sets up the Monte Carlo thread execution on all available cores passing parameters via stdin just like in the Kirin example. For the sake of the example, lets say some of the Monte Carlo threads have a state where they need to spawn off additional simulations with slightly different parameters to complete their simulation. When these threads reach this state, they can initiate thread execution using their local instance of the unified execution model. It will find nodes that have either already completed their simulation or those who have a lower load average and schedule the new threads there. Alternatively, the user can provide parameters specifying they want the execution to completely closer to the initiating thread and the execution model will try to schedule the thread on the same node, or on a node which is located closer from a network point of view.

Discussion

The examples in the prior section map out a set of experiments to explore various interface and interaction models for the unified execution model. They are not intended to provide a complete set detailing the scope of its functionality, but serve as a jumping off point to evaluate and extend the ideas.

Another thing we didn't talk about at all as how this relates to the aggregation infrastructure discussed in the MTAGS paper.

And Now For Something Completely Different

2009-05-19T17:22:00.003-05:00

I realize that I don't post much here, part of the reason is that I tend to like to post prepared ideas or results. The problem is that I then end up writing rougher ideas in some other notebook where they get lost (either physically or in the deluge of similar notes). For the longest time I was keeping these notes on my tablet PC in ink form which was a natural and convienent way to organize my thoughts and do illustrations without much thought.

However, recently I've switched to the macbook pro as my primary platform and so have more or less left the tablet behind and now find myself blogging in files using ACME, or in private Plan 9 wikis, or even within source control systems. The problem is these different sources end up being difficult to search and correlate. Since almost everything I'm doing is now publically funded, there is less reason to keep anything confidential.

So, in the spirit of transparency I'm going to start trying to use this blog to keep my thoughts and rough ideas.  More refined versions which come out of these rough thoughts will make there way to the Inferno Programmer's Notebook (if appropriate), my Linux.com blog (in particular for v9fs bits), or my DeveloperWorks blog (for IBM specific stuff). For more personal stuff, things tend to trickle through on my twitter.

I'll try to keep things tagged appropriately to help manage the signal/noise ratio for folks looking for specific bits (and also to help enhance my searchability.

quick idea: plumbing twits

2009-04-16T14:24:00.003-05:00

Because I own a tablet, I use evernote as my online notebook to organize thoughts and whatnot. They introduced a twitter bot this week that allows me to twit notes to my Evernote notebooks. It struck me that this is kind of the tip of the iceberg -- I'd like to play with the idea of using plumber(1) on Plan 9 to route tagged plumbed messages to a uBlog, Blog, Wiki, Todo list, or whatever. I suppose this might be obvious to some people, but I think it would be really useful.

public git hosting services

2009-04-01T20:29:00.004-05:00

I've been using git since its inception for maintaining the v9fs kernel module. For personal projects and branches I was using my home server, but my cable modem (and my sysadmin skills) aren't always that stable. For my Inferno branches I started using google code, but what I really wanted was the ability to create private branches (and have others create branches of my work), but still have the ability to track them in some way.

Today I checked out two public git hosting services (well, public for open source projects) -- gitorious and github.
While gitorious seems a bit more open, and allows private forks while maintaining links -- githubs integration with code review makes it a bit more preferable to me. I'm thinking of moving the v9fs development branches to it and just using the kernel.org git repo for my upstream branches. The idea being if others want to branch my repo to add functionality or fix bugs I haven't had time to get to, this should make it easier.

github is also linked to lighthouse, which provides bug tracking and milestones. We've been using kernel.org's bugzilla for v9fs, and while no one seems to use it besides me, its fine and that way it links in to the overall kernel bug statistics. The lighthouse approach also can be used to track milestones and component feature implementations. I've tried to track roadmaps and whatnot in the past using launchpad -- but its lack of git integration made it ultimately more annoying than useful. Its not clear to me that lighthouse will be any better, but I'm going to try it with a pet project for now and maybe switch v9fs milestones over to it later.

Plan 9 Commentary

2009-03-26T18:53:00.003-05:00

It's a real shame there isn't a good tool in place to help update (and keep up to date) code commentary like Nemo's. Such commentary is really a great resource to refresh one's memory or learn about the system -- it seems like there should be a way to link the commentary to something more similar to the unified diff format instead of line numbers -- of course that is only half the problem -- if no one goes through and updates the data its pretty useless. I wonder if it would be worthwhile to transcribe Nemo's comments inline using a light-weight literate programing style -- that would certainly help encourage folks to update the commentary when changing the code (and moving it around respectively) -- of course it would also piss off anyone that doesn't like literate programing which often makes it so you can't see the code for the comments. It strikes me that the various code review tools have nice hooks to commentary, but I'm not sure these are kept linked in the main tree (or just in the patch).

Along the same lines, we are far overdue for joining the podcasting bandwagon -- I've spent the past couple months using my gym time to take in lectures from stanford, apple, nvidia, google, and others all on my iphone. It would be really nice to have a set of videos for the Plan 9 community on everything from getting started to kernel development. At the very least I'm going to make an effort to cover my existing and future work with such video podcasts as an alternative, more effective way of getting information across. Time permitting I'd like to do the same thing for all components of the system, perhaps stepping through Nemo's walkthrough as a model.

srv² - next generation service registry

2008-12-03T11:34:00.003-06:00

The srv(3) device seems like the ugly stepchild within the Plan 9 system. In a system focused on distributed resources, it is a node-specific service registry. Within Plan 9, its a node-global resource and doesn't allow for multiple instances which can be placed in different dynamic private name spaces. Device resources are treated separately, having separate mount shortcuts instead of being integrated into the srv(3) registry. In common practice users must rely on static configuration files to access network resources and post them in the local srv(3) instance. This static nature relies on alternate layers to provide reliability and scalability for resource access.

In order to explore potential solutions to these issues and inconsistencies I've been thinking through some ideas for extending the srv(3) concept to be more useful in a distributed environment and allow better integration with core Plan 9 principles. In all cases I'll likely implement it as a synthetic user space file server which will use the normal srv(3) as a back-end. Add a concept of scope to the service registry -- right now there is a single level of scope on Plan 9 (node-level). Inferno has multiple srv instances via attach arguments, but nothing that really amounts to scope.

The idea is to allow several automatic levels of scope as well as user-defined scopes:

Program Group Level
User level
User group level
Node Level
Tree Pset
Immediate Neighbors (in Torus)
Cluster level
Social Network (Community Level)
etc. (you get the idea)

The idea of automatically propagating/discovering neighbor services and cluster services is linked to ideas resulting from our port to Blue Gene as part of the FastOS work -- but also is related to previous ideas I've had related to collaboration services.
When one registers services at scopes larger than node-level, the local srv² communicates with appropriate neighbors to make sure their registry for that scope is updated. In this way, services are immediately available in the other registries. I'd like to explore both the push and pull methods for this update, a lazy pull/poll may be more appropriate than a push. One thought was to support a plug-in style architecture for this allowing control of different scopes to be maintained by different applications. This could allow for different membership/authentication schemes as well as different discovery. In such an approach srv² would actually only be responsible for overall management of the hierarchy.

In order to keep things tidy, there are some additional nit-picks with srv(3) which must be fixed. In order to organize the various scopes and the larger potential set of services I'd like to make the srv² registry a deep hierarchy. This may just be used for scope or may be used for additional levels of organization. I'm going to start simple with the top layer identifying scope and the second level being the services associated with that scope. However, I'm going to structure the implementation to allow nested scopes and organizational constructs.

Another nit-pick I've had with srv(3) is the lack of the ability to use group permissions. One of the scope concepts is to allow group scopes, so in order to support this we'll need to incorporate group management into the srv² infrastructure. There is an unfortunate amount of complexity here. Groups should have scopes as well, allowing node local groups as well as multiple levels of cluster groups. It would be nice to allow user defined groups instead of just system defined groups -- and this sounds more like ACL's rather than traditional UNIX groups. I'm not confident on a path to pursue and would welcome suggestions.

The requirements of our FastOS research include reliability and scalability. For cluster based services it would be nice to integrate support for load-balancing and fail-over directly into the srv² registry. If we use srv² in a similar way to how we use srv(3) then we'd only be able to load-balance or fail-over at mount time (at least without adding an additional layer). An alternative would be to provide an auto-mount feature which would attach to the service on crossing the mount point and automatically load-balance or fail-over the service as necessary. There is a large degree of complexity here, and an inherent danger in doing anything automatically. Still, for a certain class of service this style of resource and application, this form of access may be preferable.

A final area I'd like to explore is discovery. ndb/local has always bothered me as being far to static. Inferno had the concept of local discovery via the virgild which would allow discovery on the local network. With cluster functionality built into srv² we just need a way for the various srv² instances to discover each other. Locally we could use a mechanism like virgild or utilize zeroconf. For broader discovery we could use central registries, perhaps utilizing Chord concepts to make it a bit more scalable. Within our FastOS work we'd also like to explore hierarchical or toroidal organizations, scope, and discovery for our service registries.

This will be ongoing work that I plan on implementing and exploring during the first quarter of 2009. I'll update this blog with my experiences, and hopefully compile a paper towards mid-year with the results of the experiment.

Cross Post: lab 90 - Multicast DNS and Zeroconf (client portion)

2008-11-10T13:29:00.000-06:00

I wrote up my experiences with adding zeroconf and mdns clients to Inferno in Lab 90.

Is OP just CRUD? Give it a REST.

2008-10-24T09:43:00.005-05:00

Background

Octopus Protocol

http://www.9grid.net/IWP9/IWP9-Guardiola.mov
(Presentation) http://plan9.escet.urjc.es/ls/export/opiwp9.pdf

(Non-technical observation is that we need to get better at filming/editing our presentations, our production values suck!)

Observation

So - what is being done in the Octopus Protocol is very similar to the current approach being taken by many web developers. The primary difference between OP and the HTTP approach is that the OP Put method provides additional metadata. You get a bit of a tradeoff in that 9P provides an authenticated persistent session, but this may make caching and other intermediary operations more difficult.

Discussion

Protocol differences aside, there are some interesting opportunities here -- for one, why has anyone tried to write web applications using a synthetic file server as the "active" backing? I alluded to this a bit earlier in my Service-Oriented File Systems Post, but understanding more about how the web applications use REST I don't see any reason why an httpd server couldn't be constructed which interpreted HTTP PUT operations against a path as write operations against a file which are followed by a read of that file to get the result (error) to send back to the user. I'm not sure if it would make sense to incorporate the service documents as something generated by httpd or by the synthetic file system itself..probably better done in the synthetic file system so you could specify types of data as appropriate.

Another interesting aspect is how Google App Engine works with these concepts, essentially you have a template file which maps URI paths to application handlers. This essentially is like a namespace specification, but it might also be useful as a method to compose synthetic file systems out of simple pieces where each component knows how to read/write (maybe just via stdin/stdout) and more advanced components could handle things like dynamic directory structures.

Finally, you could think of a new multi-faced model for UI development which in some ways is similar to the O/mero approach. In one "view" of the namespace you create directories and files representing the various UI elements -- included in their descriptions are attributes defining their characteristics, types, and position (most likely hierarchical and not absolute) -- another "view" of the same namespace would export html (and javascript?) elements comprising the form which satisfies the specification made in the other view (this could then be skinned with CSS to make it pretty), and the final view of the namespace is an active synthetic file system which would accept the file reads/writes from the HTTP put operations against particular elements of the form. What you end up with is essentially a shell/namespace/filesystem SDK for building web-apps.

Venti and Linux

2008-08-26T16:44:00.003-05:00

I've been playing around with the plan9port's version of Venti in a bit more depth lately and thinking about Linux-specific optimization opportunities. I've got more thoughts than I'm prepared to write down right now, but I figured I should start somewhere.

First, there seem to be a number of shortcomings for large systems. We've actually just installed a new 36TB storage system here at work with 3 file server nodes connected to it with 64GB, 64GB, and 12GB of DRAM respectively. One of the first things I ran into is Venti seems to behave poorly if you try to give it too big of a mem, icmem or bcmem (i was going for 4GB icmem and 2GB bcmem and mem) -- I'm fairly certain something is failing silently, but haven't had a chance to track it down. Another potential shortcoming on these 16-way, 16-way, and 4-way systems is that venti appears to be single threaded at its core so most of the processing power of these systems is idle while performance suffers.

Most of my current experimentation revolves around looking at using venti to backup and back block devices. vbackup does a reasonable job of this, but the fact that it scans the entire volume is a little problematic (my 80 gb home directory takes 45 minutes to go through even if I haven't modified anything). I'm currently looking at using lvm2/device-mapper to take snapshorts to make backups more coherent and to track changes since the last venti so I only have to have venti operate on changed blocks. This should allow much tighter granularity on dumps than nightly.

Of course, there is little need to keep 5-minute, 15 minute, or even hour level granularity snapshorts forever. I've started to think through a hierarchical venti approach which used a local 'transient' venti which would use a recycled arena for the transient snapshots and then using venti/copy to send coarser granularity snapshots to the next level of the hierarchy (which could be another cache server or be the central venti server).

Besides using device mapper for helping to take snapshots -- it could also be used to create caches and or cow devices on top of a venti score -- assuming I can get something to look like a disk or a disk image that actually serves a vbackup score as a disk.

More later...there's a bunch of stuff I need to play with to understand how the various pieces I'm thinking of interact.

Plan 9 Authentication in Linux paper available

2008-08-10T12:23:00.002-05:00

Ashwin's paper on implementing Plan 9 authentication and capability device for Linux is now available for free from the ACM Archives along with the rest of the special issue on the Linux kernel that I helped co-edit:

http://portal.acm.org/toc.cfm?id=1400097

9p/virtio Slides Available

2008-06-19T19:15:00.001-05:00

The slides I presented at KVM forum on using 9p over virtio to provide a paravirtualized file service are available on the KVM wiki

trans_fd_simple

2008-04-16T14:44:00.002-05:00

Did some really basic performance checks on trans_fd_simple compared to the older code and comparing single lock with reader/writer lock protection on the socket. Turns out my thinking with reader/writer lock was flawed and there are race cases (hence only the single threaded result). Long and short of it appears to be sort of what we figured, that the simple approach does better on single threaded workload due to less overhead, but does much worse on multi-threaded due to added latency and what not. I ran against npfs as well just to check if earlier thinking about whether or not multi-threaded npfs added much -- it appears to add nothing but overhead in this case.

This particular case was from a qemu hosted linux talking to the simulation host server -- which isn't particularly good. I'm going to replicate this test on the blade regression cluster, but wanted to do a quick check before moving on to scatter/gather implementations.

Service Oriented File Systems

2008-04-16T13:46:00.004-05:00

Its fun to abstract and think about web2.0 concepts in terms of Plan 9 concepts. Abstract away the gorp that makes up their implementations and think about some of the fundamental concepts. When interfaces are files, mashups are binds.

The larger picture here is about keeping this simple, language independent, and client independent. I'd like an infrastructure that I could build simple web services out of that can be viewed with a browser, or with a rich-client (like acme plug-ins). I'd like the front-end of these services to be defined in the same way the Octopus approach defines normal GUI's (except from a web-browser, the widgets are implemented by Java script). I'd like the back-end of these to be akin to big table and the chubby lock service, with a back-back-end that is simply Venti versus a database or XML.

I'll admit I'm probably naive about all of this, not having really worked on client-facing applications for some time, but I think it would be a fun thought exercise to go through and perhaps a nice refresher to actually try and implement. While part of the goal is to keep things language independent, I'll probably do things under Inferno to keep them portable for me and open up the potential for interaction with the Octopus crowd.

Of course, this all falls under the copious spare time which I have a definite deficit of -- however, I think dedicating one day a week is doable and I should be able to combine this with my constant desire to get back and look at collaborative facilities (aka warren) under Plan 9 and Inferno.

v9fs updates

2008-04-14T09:33:00.002-05:00

With the 2.6.25 merge window rapidly approaching, it looks like I'm going to have to hold off on merging the reorganization until 2.6.26. At the moment, I'm still fiddling with the transport API and specifically the rewrite of the file descriptor based transport which is used for sockets, named pipes, etc. With a 2.6.26 target, I should be able to get the async and scatter-gather rewrites complete as well as merge the in-kernel server code from lucho.

As part of the exercise I've created a "simple" version of the code which is synchronous and single threaded (to match spfs). My motivation was to scale back and figure out what the core pieces were in the fd transport, as well as to have a useful performance baseline to measure the overhead (and advantages) of a more asynchronous model. A nice side-effect is that I also have a really simple and straightforward transport module which can be used as an example for those wanted to write their own transports. I'm not quite certain whether I want to merge the simple transport mainline though -- although I may merge it into trans_fd in order to maintain it as a baseline case -- it really amounts to a different request member function which could easily be selected via an option.

In parallel I've setup a few blades in order to measure performance, and perhaps more importantly performance regression. When complete, I hope to publish historical performance regression results back to kernel 2.6.14. At the moment I'm trying to do all collection of results and graphs inside Google docs, which should make an easy place to keep things up to date and simultaneously publish the data. I need to look around to see if anyone has hooks for the Google Data API so I can have my regression framework update the results automatically instead of me having to go in and manually enter the data.

GSoC 2008 Ideas

2008-02-26T08:27:00.004-06:00

My Plan 9/Inferno summer of code ideas (otherwise known as projects I wish I had more time for) - projects should take no longer than 5 weeks to complete (including ramp up, debug, and packaging) -- conservatively projects should be completable in 3 weeks by experienced P9/Inferno developers and 2 weeks by folks unfamiliar with p9/inferno development environments.

mount.9P helper program for v9fs including packaging for debian and fedora -

hueristic for determining transport type

DNS support for resolving hostname <->ip address
ssh-based tunneling support (will require server work as well)
man pages
debian packaging
rpm packaging
(stretch) authentication support (w/p9p)
(stretch) authentication support (w/o p9p)
(stretch) integration with idmap solution

since this is relatively straightforward it will really need to be "super" version including support for ssh tunneling, authentication, etc.
idmap solution for v9fs - maps local uids/gids/error-codes to strings

userspace daemon to provide mapping
hooks in v9fs to use mapping
(stretch) synthetic file server approach to update mappings

OLPC Inferno Environment

fontfs - on-demand ttf solution
metafs - abstract metadata from underlying file system
(stretch) Integrate OLPC translation solution for Inferno

(stretch) OLPC oriented GUI toolkit, window manager, and toolbar
(stretch) Inferno approach to collaboration using file systems

Source Control Management based on Venti backend

single branch version tracking with associated log file

add new files/directories
commit changes
checkout specific version #

(stretch) support multiple branches
(stretch) support repository sync (push/pull)
(stretch) support three-way merge

wrapper for p9p vbackup to make it more user friendly

wrapper which tracks venti scores based on volume being backed up
GUI admin tool

which sets up venti in partition(s) or with files

which assists in configuring backup intervals and volumes

(stretch) time-traveler like GUI for navigating/searching backups
(stretch) support for pruning and/or merging snapshots

GSoCFS for managing future community involvement with GSoC

synethtic file system and web interface
posting project ideas
voting for projects
registering student interest in projects
project milestone tracking
project blogs and wikis
post-summer project success metrics (subjective and objective)
(stretch) syndication points for community monitoring
(stretch) potential integration with SCM
(stretch) potential integration with some form of chat
(stretch) potential integration with name space sharing

(more to come...)

metafs

2008-01-28T19:43:00.002-06:00

Since I wrote about it in my other post, I might as well explain metafs. Basically in order to accommodate the OLPC file system security model I need to maintain uid, gid, and mode bits for Inferno's view of the underlying file system separately. This is actually a common need within hosted Inferno when you don't necessarily want to keep Inferno user's sync'd with system users. In any case, this amounts to an overlay file system, which stores away the additional metadata.

While I am tempted to keep a central database of the metadata, the most pragmatic solution would seem to either use extended attributes or a hidden file in each directory. Since extended attributes are somewhat distasteful and not available anywhere, I'm going to start with the per-directory hidden file.

In the future this same mechanism might be extendable to implement semantic file system extensions (or extended attributes for that matter) -- allowing for tagged organization and searching of the file system as well as alternative views such as a "journal" style temporal view (which might also be useful for backup purposes). However, these extended metadata scenarios may beg a re-evaluation of the distributed versus centralized storage model.

In any case, the basic plan is squirel away the extra meta data, indexed by the string filename/directory-name in a hidden file. If the directory isn't writable by eve then no extended metadata will be written and the directory will default to the underlying uname/gname/mode -- however, if eve isn't able to access the files, the overlay won't be able to access the data -- so it's likely a moot point.

freefontfs - a styxserver inferno-lab

2008-01-28T19:33:00.000-06:00

I've decided doing the dynamic free font file system would be a more gentle introduction to using the new Inferno styxservers modules than a full-blown filterfs like what we plan for metafs.

The most appropriate example would appear to be mntgen.b (which allows for dynamic walk based creation in /n). It uses both the styxservers facilities and the navigator and has a dynamic nature similar to what I want for freefontfs.

Fonts are comprised of two types of file, the base file (..font) and subfonts (..). We only want to generate (and cache) files as they are accessed. The base available 's will be derrived from crawling a directory for ttf files or perhaps from a configuration file.

Following mntgen, it looks like it would be much more simple to keep it as a single level hierarchy (much like /n) -- this may be possible as all the necessary information is encoded in the filename. We just need to parse the single walk correctly.

One question is whether we should prepopulate the cache with a base font size for every ttf available. This seems like a reasonable idea, but probably wouldn't even be necessary in many situations -- perhaps we can leave it as a config option.

inferno-olpc circa 1/28/08

2008-01-28T09:26:00.001-06:00

This week's update to inferno-olpc includes OLPC-optimized fonts and a customized logon with two extensions:

it binds the Inferno activity "writable" area to important areas of the Inferno name space which the user may want to write to (/usr, /services, /keydb, /tmp). This will be necessary for the next OLPC update which will start enforcing security (including only allowing writing to the activity writable area of the file system. Also bundled is a day-one scenario which will setup template directories to bind from and copying relevant files.
when logon can't find a user it will now prompt to create the user by copying the template /usr/inferno directory to /usr/username. This can probably be generalized back into the standard Inferno base.

I also have a "customized" toolbar which just limits the size of labels to 15 characters so you don't get the awful line wrap. This is a stopgap, a better toolbar metaphor needs to be developed for the OLPC version of Inferno. I also augmented wmsetup with a larger set of applications.

I had a larger agenda that included social-network style "profiles" and integrated security/password setup. But there were too many corner cases to test and "bullet-proof" so I decided to hold that off for later and maybe just incorporate those functions into separate applications.

I'm still a little uncertain of how I want to handle authentication and security. On one side, there could be a central "schoolserver" to setup authentication against. However, as most of the current users don't have such a server, having a pure peer-to-peer mode is probably preferable. Given that every laptop will be an auth-server then, do we setup a laptop wide authentication setup, or do we do a per user authentication setup. The later would seem to make more sense to me as the user-provided password wouldn't have to be stored anywhere and could be use to setup keyfs and the user's own authentication secrets. Other users "registering" with that user for the purpose of collaboration could then provide their own secrets in his keydb, etc. I'm not sure if that makes sense or if there is a better model I should be considering.

The file system security is another pain. Inferno users won't necessarily equal underlying system users (a common problem). This will be made worse with the new OLPC security enforcement which may have different uid's for each "instance" of Inferno (but the same "gid"). I think the right thing to do is a new version of devfs which always keeps group permissions open in the underlying fs and uses extended attributes or some other mechanism to maintain Inferno uid's and gid's for the files.

Inferno Enabled

2008-01-23T08:54:00.000-06:00

I've resurrected the Inferno Enabled sticker logo.

PNG Version:
SVG Versions are available in white and black.

Infernal OLPC repository

2008-01-21T16:38:00.001-06:00

I've reset up my Inferno git repository on http://git.9grid.us/git based on a mirror of Forsyth's SVN repository. Included now is my working OLPC modifications repository which contains both my core-Inferno patches as well as new mk rules for packaging. I'm going to move on to better OLPC applications support now, but there probably won't be an update on that until after next weekend.

Infernal OLPC Packages Available

2008-01-14T08:53:00.001-06:00

Finished working on the details on how to launch Inferno within the OLPC infrastructure and packaged an initial release which is now available via the OLPC wiki.

I need to take the sandbox I built it from and work out a merge into the mainline inferno-os repository. Basically this will amount to moving some bits into olpc specific directories and writing a distribution script which builds the bundle.

Next step after that will be to really go to work on customizing the environment for the OLPC, starting with a custom look and feel for the window manager and a few example applications. That will include supporting scaled true-type fonts and perhaps SVG support. I also need to work out some day-one scripts to help setup the right name space for the OLPC's restricted security model.

OLPC - Hosted Inferno

2007-12-31T18:09:00.002-06:00

Got my daughter an OLPC for Christmas, but have spent a lot of time playing with it myself. First order of biz was to load Inferno on it ;) I can run it from the shell quite happily, but am now trying to come up with a Sugar launcher and package for wider deployment. I may start with a simple sugar app that toggles between loading Inferno versus loading sugar. This should be much easier as it'll avoid interacting with the Python gorp. Another alternative would be to just hack the terminal application (which would then act as emu's console), but I'd like to try to find a way to tie it in better.

(UPDATE: launcher is working, now working on packaging)

Service Oriented Synthetic File Systems

2007-11-30T08:52:00.001-06:00

I was doing some thinking about how larger synthetic file systems are structured and implemented under Plan 9 (or using 9P in general).

My toy-use-case for thinking about it was developing a synthetic-file-server based service similar to SourceForge -- where you have many different projects, each with different component services (bug-tracking, version-control, blogs, wiki, forums, etc.)

You could do it by running lots of different servers (per project) and binding them into a name space and then exporting, but that seems kinda awful (particularly if you have dozens of projects you are using the same tool to track). Much better to have a single server running which is able to accommodate multiple projects, each with multiple file services underneath.

Obsession with qid-spaces ends up being distracting, with the way fids work, we shouldn't need to segment the space so rigidly, it seems to me that qids only really need to be unique per directory, outside of that you can track which "module" of the hierarchy you are in by saving some state in the server-side fid tracking structure as you traverse the hierarchy.

dot-dot makes things a little more complicated, but the solution used inside the kernel can be used within file servers as well (namely, keep the whole path in the service-side fid tracking structure).

What's really nice is you can develop plug in file server modules which allow you to compose much more complex single-server-services out of many different file system components -- without worrying about how the qid space is split up and managed.

Applying mount namespaces (in Linux)

2007-10-03T14:56:00.002-05:00

Decent developer works article describing how to use private name spaces in Linux including the new shared subtree stuff and examples about how to create per-login or per-user (where's per-process?) name spaces.

"Applying mount namespaces" IBM Developerworks

Per-process would be fairly easy to do given the techniques described in the paper -- we just need to work through an example, although it may be easier to do inside the source code of a particular shell.

v9fs performance versus NFS

2007-08-25T20:55:00.001-05:00

I've been doing experiments looking at using 9p between partitions for file system access - focusing on optimizing a shared memory transport and eventually a cooperative cache.

However, I ran some baselines between partitions using the virtual network and it seems our performance has slid a bit.

Here's the results from 2 years ago using a gigabit network between two hosts:
And here is the result between two partitions on KVM comparing NFS, 9p, and local virtualized disk: (9p cached is loose cache mode, but since the working set is double the memory of the partition this actually worked against it):

Put block and get block are equivilent to the write_block and read_block of the older graph. Where 9p used to edge out NFS on reads, we now are a good deal worse. Our write performance looks like its as bad as it used to be and rewrite is worse.

Overall, we've had a big slide -- there has been two rewrites of the transport and support for multi-threaded workloads (which this workload doesn't exercise). This is also against a different server than original.

I need to go over my methodology a bit, but I think its probably worth revisiting performance issues.

Blue Gene Project Pages

2007-06-22T14:36:00.002-05:00

I've setup an more official website to put publications and presentations relating to the DOE sponsored Plan 9 on Blue Gene work. You can get to it here and I'll be updating it over the next few days.

Torus and tree networks working on Plan 9 BG/L port

2007-06-18T15:15:00.002-05:00

We finally worked out the remaining infrastructure issues and are now able to cpu(1) to both I/O nodes (over the Ethernet) and cpu nodes (over the tree network). The cpu nodes are also able to talk to each other over the torus network.

We will be attempting a live demo during the USENIX poster session, so drop by and play with Plan 9 running on a Blue Gene.

Inferno Running Byte Code on BG/L

2007-06-18T15:10:00.002-05:00

Inferno is also running (hosted) on both Compute and I/O nodes.

9p and Linux Articles

2007-05-11T15:48:00.002-05:00

By request, I've posted the two "semi-complete" drafts of a 5 paper series that intended to describe using and programing with 9p under Linux using v9fs and spfs. These are very fragmented drafts and somewhat out of date. If anyone is interested in finishing them off, please contact me. The original idea was to publish them on IBM Developerworks.

CPU working on Blue Gene

2007-04-20T20:35:00.001-05:00

It took quite a bit of fussing about to gateway between the IBM internal Plan 9 cluster and the Blue Gene VLAN, but with the help of Forsyth's new Ethernet driver for BG/l, Inferno as an intermediary on the front-end node, and a few other bits and pieces... we were able to cpu(1) into Blue Gene I/O nodes. The CRN-Tree network is pushing packets back and forth and the Torus is being debugged. Overall its been a very productive 80-hour week.

two cores walk into a bar

2007-04-17T21:41:00.001-05:00

So, we are bringing up Plan 9 on the Blue Gene supercomputer and stuff looks great, we boot it to a prompt, we can interact with the machine. Its just every so often we see some corruption in the output, and then a little later on the thing dies with some form of panic or critical exception. Shit... We think we have some weird memory corruption problem and go on a witch hunt through the code. So, two bits of additional technical information -- our output is obtained by dumping the console prints into an SRAM and then sniffing it with a JTAG. The SRAM is shared by two non-coherent processor cores. We had assumed the one core was held in reset during boot.

We were wrong.

The freaky shit is, the damn OS booted to a prompt and I could type, list directories, screw around -- and we were getting a single output stream that was more or less coherent. The cores were executing in such precise synchronization (in the same memory space no less) that they even were writing the same characters to the console buffers and incrementing the console pointer to the same value. Complete lock step.

We wouldn't have even noticed except that we started doing the ethernet and touching it started skewing the two processors, and we started getting ddoouubbllee ccoonnssoollee oouuttppuutt and then they would go back into sync.

Its amazing how really, terminally, completely broken shit can run for a damn long time...

More Plan 9 on Blue Gene

2007-04-12T10:03:00.001-05:00

For folks who can't read the memory dump in my last post I've got a rdbfs(4) like interface now that gives me access to the BG/l machine state and memory and includes a more readable console interface:

criswell% con -r /usr/ericvh/bgd/0/con0

Plan 9 bgl
cpu0: 0x5202

BG/l Personality
Block: R000-N60_32_4
Memory Size: 536870912 bytes
clockHz: 700000000
Torus Addr: -1 -1 -1
Tree Hops to Top: 255
EMAC h/w Address: 0:d:60:e9:10:9f
Assigned IP Addr: ac186439

Kernel Status Cpus 0 & 1: 3 3
512M memory: 160M kernel data, 352M user, 977M swap
boot...
--rw-rw-r-- c 0 24 Apr 11 2007 /dev/bintime
--rw-rw---- c 0 0 Apr 11 2007 /dev/cons
---w--w---- c 0 0 Apr 11 2007 /dev/consctl
--r--r--r-- c 0 72 Apr 11 2007 /dev/cputime
--r--r--r-- c 0 0 Apr 11 2007 /dev/drivers
--rw-rw-r-- c 0 48 Apr 11 2007 /dev/hostdomain
--rw-rw-r-- c 0 0 Apr 11 2007 /dev/hostowner
--r--r----- c 0 0 Apr 11 2007 /dev/kmesg
-lr--r----- c 0 0 Apr 11 2007 /dev/kprint
--rw-rw-rw- c 0 0 Apr 11 2007 /dev/null
--r--r--r-- c 0 0 Apr 11 2007 /dev/osversion
--r--r--r-- c 0 12 Apr 11 2007 /dev/pgrpid
--r--r--r-- c 0 12 Apr 11 2007 /dev/pid
--r--r--r-- c 0 12 Apr 11 2007 /dev/ppid
--r--r--r-- c 0 0 Apr 11 2007 /dev/random
--rw-rw-r-- c 0 0 Apr 11 2007 /dev/reboot
--rw-rw-r-- c 0 0 Apr 11 2007 /dev/swap
--rw-rw-r-- c 0 0 Apr 11 2007 /dev/sysname
--rw-rw-rw- c 0 0 Apr 11 2007 /dev/sysstat
--rw-rw-r-- c 0 78 Apr 11 2007 /dev/time
--rw-rw-rw- c 0 0 Apr 11 2007 /dev/user
--r--r--r-- c 0 0 Apr 11 2007 /dev/zero
/boot/bind
/boot/boot
/boot/echo
/boot/ls
/boot/ps
/boot/rc
/boot/rcmain
/boot/sleep
Hello Squidboy
sleep 10
hi hi hi hi hihihihihihihihihihihhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh
%

(Picture photoshopping credit: Andrey)

Plan 9 Booting on Blue Gene/L Hardware

2007-03-30T15:27:00.000-05:00

This is just the 1st I/O node, but the other 3 I/O nodes and 32 Compute nodes in my block were in the identical state.

mmcs$ {0}.0 dump_memory 0x808d9210 0x1000
OK
{0}.0 Dumping Memory
0x808d9210: 0d0a506c 616e2039 2062676c 0d0a6370 ..Plan 9 bgl..cp
0x808d9220: 75303a20 30783532 30320d0a 65746865 u0: 0x5202..ethe
0x808d9230: 72303a20 6e6f2065 74686164 64722069 r0: no ethaddr i
0x808d9240: 6e204545 50524f4d 20656e76 0d0a6574 n EEPROM env..et
0x808d9250: 68657231 3a206e6f 20657468 31616464 her1: no eth1add
0x808d9260: 7220696e 20454550 524f4d20 656e760d r in EEPROM env.
0x808d9270: 0a323536 4d206d65 6d6f7279 3a203834 .256M memory: 84
0x808d9280: 4d206b65 726e656c 20646174 612c2031 M kernel data, 1
0x808d9290: 37324d20 75736572 2c203539 304d2073 72M user, 590M s
0x808d92a0: 7761700d 0a626f6f 742e2e2e 0d0a2d2d wap..boot.....--

Virtual Systems

2007-03-18T17:24:00.001-05:00

Bit the bullet and got Plan 9 running under Xen on my "victim" laptop. It can now run Plan9 natively, under Xen, or under Qemu. I still am toying with the idea of a bridged network mode where I can mount an Inferno devip from a loopback cable to my windows laptop in case I can't find hard-lines to plug into at Watson. This also means my victim laptop is squared away to be a testbed for x86/Libra -- I might even get to a basic "shot-in-the-dark" (ie. no Xen devices, just devshm) configuration sometime during the trip.

xsm support in

2007-03-14T17:35:00.001-05:00

I've added the Xen shared memory channel bits to devshm and pushed the code to the inferno-power repository (although hypothetically this is a more generic feature). I really need to separate the devshm code from the power code, it just seemed more convenient to work on them in the same branch.

The Xen shared memory code only functions as a server right now. I just tested it with a slightly modified Libra client and everything seems cool. I need to revisit performance results to make sure I didn't screw anything up and work out the right way to modify the Libra launch scripts.

devshm updates

2007-03-10T15:12:00.001-06:00

Spent half the week chasing down a truly ridiculously stupid bug in the mmap code for devshm. I had intended to have the Xen based shared memory done by now and maybe even a Libra version of Inferno -- unfortunately that has all gone down the tubes for the time being.

The devshm code with SysV and mmap based shared memory transports is available off of the 9grid.us GIT repository, I need to write a man page before submitting a patch to the Inferno mailing list. Now I'm off to port the xen devchan interface.

Linux-Power Inferno (take 2)

2007-02-28T19:14:00.000-06:00

Nearly complete with a second take on my Linux-Power Inferno port. I've corrected (with Forsyth's help) many of the hacks I had rushed through the first time. I'm still a bit concerned with floating point and math libraries, but on the whole it seems functional and no longer uses pthreads so hopefully the SMP problems we were seeing will go away.

I'm trying for a clean merge against the Google Code inferno-os tip now, which should leave me plenty of time tomorrow to do a clean integration of the shared memory transport with shm, mmap, and xen back-ends.

Collab

2007-02-21T20:42:00.001-06:00

In lamenting not really knowing the various states of a distributed team working on Plan 9 related projects, I was thinking through expedient mechanisms for loosely keeping in touch. There are a bunch of tools that seem to help with distributed teams: phone calls, IM, IRC, VNC, Peer-to-peer and centralized file sharing, SCM, etc. I was thinking of architecting a simple suite of tools to provide basic fuctionality within a Plan 9 paradigm.

The cornerstone would be a modified version of faces, which checked against a centralized file system (or maybe just a centralized place with file systems bound from several different servers). People currently active ('logged into') within the collaboration framework would have their face appear. Mouse-over the face gives a quick bit on their current status (i.e. working on /sys/src/collab.c), right click pops up a menu of different options (chat, blog, view, share, mount, etc.) -- each of these would plumb an action to an appropriate tool (either in ACME or raw).

chat - simple point-to-point IM facility, I'm toying with the idea of walkie-talkie style audio chat as well.
blog - pops up the person's "blog" which really is just news(1), but personalized for that person, maybe work out something more complicated with wikifs later, but news(1) will do for now
view - If the person has shared a window with you, view will plumb it to a viewer -- I haven't quite decided how to handle different types of views, could just do VNC, but I'd rather have the ability to share windows -- or really what I want is the ability to share ACME windows and keep the sides in sync (like a distributed Zerox) -- ultimately, there's all sorts of granularity to sharing, but for now it'll be a single thing (kinda works like snarf) and it'll have to be integrated into ACME and other tools which use it.
share - really mostly described above -- this basically grants the person access to your current share
mount - pops up a window with the person's public namespace mounted -- this could just be their /usr directory, or it could be something more. Haven't quite decided how to handle this completely.

There will be a variety of command-line tools that allow you to post shares, export portions of your namespace into your public namespace, and allow you to easily update your blog or status.

I think the simple approach works for 1-to-1 communication, not sure how to handle 1-to-many or many-to-many without things getting complicated. There's also various ways you can build on this to add additional context (like maybe per-project collaboration spaces). They key thing is that everything works through file system mounts.

A big missing item is SCM, I have been toying with some ideas around a git-like mechanism with a venti back-end. For the most part this isn't an immediate requirement though, centralized file repository backed by venti is fine -- although it would be nice to have a formula or script for setting up replicas of such centrailized repositories.

9p: loose cache

2007-02-21T20:37:00.002-06:00

Loose cache mode got sucked into Linux 2.6.21-rc1 -- at least the read portion of it did. You can now specify cache=loose as a command line option and v9fs will attempt to aggressively make use of the dcache and page-cache. This cache is complete non-coherent -- so I'd advise only using it with exclusive read-only mounts (like vacfs). I've got a version of the writepage stuff (and it is in the -mm kernel), but I'm currently reworking it to make it more like NFS. I've got a couple more optimizations in the pipeline that I can use with the loose cache, such as pre-populating the dcache during a readdir, and a general reduction in the number of stats sent during transations.

A bigger change on the horizon is the introduction of support for async i/o and read-ahead. These should allow us to match (or beat) NFS speeds while in loose mode. Adding some level of coherency to the cache shouldn't be too difficult, but I don't have a driving application right now, so it'll stay low on the priority list.

v9fs writepage woes

2007-02-14T19:35:00.002-06:00

Been struggling with the writepage implementation in the loose cache mode. As a result, I've made get_attr loose (it will just use info in an inode versus requesting it from the server) -- as a side-effect, Bonnie rewrite performance doubled and I'm not setup to pre-fetch inode info during a readdir (which should make ls -l much quicker). FSX, Bonnie, Postmark, and Dbench all pass. Multithreaded dbench gets hung up sometimes during cleanup -- not sure why. The addition of writepage makes mmap writes possible as well, however, FSX fails when I enable the mmap write tests (with a data corruption error) -- this is likely a similar problem with the inode not being properly updated.

Otherwise, I've addressed most of Andrews points. I don't think I'm doing prepare_write quite right yet -- I guess I need to look at locking the page down. I also have to make sure I go through and take out any redundant code I put in for debugging. I need to draft my OLS proposals tonight, so it'll have to wait for tomorrow. I think I'll also take the extra steps to disable mmap writes for non-loose items -- also should probably disable mmap reads for synthetics.

Plan9/Inferno Revision Control

2007-02-14T10:53:00.001-06:00

I've been playing with the idea of some simple shell scripts that would provide git-like-service on-top of venti for use as wide-area source control. Essentially, to access a repository, you'd need access to the venti hosting the repository and the current score of the repo which could be published in a 9p partition or web page or whatever. Alternatively you could have the branches of the repo published (or pull the branches from the single score).

The scm-part of the repo would contain the branches, tags, and log messages. Branches and tags are just files with the particular score in them (not sure that there really is a difference between the two) -- might also squirell a description or something in the file.

Log messages would be in files named after the score of the tree that their commit represents. The log message would contain a parsable pointer to the parent score of the commit. Probably reasonable to organize the log messages in directories by date or something to keep things efficient. The logs themselves are probably best stored in venti, referenced by a "special" branch named log.

Then it's just a matter of writing some scripts to handle initializing a scm database, adding and removing files, commiting changes, and helper routines for doing diffs, merges, branching, etc. Also could have git-pull/git-push equivilent commands for syncing remote and local venti stores (if you wanted to have a local venti).

Interesting collective viewpoint on the future of computing

2007-02-11T10:15:00.001-06:00

http://view.eecs.berkeley.edu/wiki/Main_Page

v9fs updates for 2.6.21

2007-02-10T00:13:00.002-06:00

Working on putting cache features back in as an option. I'm basically thinking of having several modes:

loose - intended for exclusive read-only file systems, dcache and page cache never revalidate
tight - intended for shared file systems, dcache and page cache always revalidate
decay - nfs-like mode where we only validate if more than a certain amount of time has passed
session - exactly like loose, but with some extra bits for use with fscache so that we can verify that the backing store hasn't changed while we've been detached

Right now I'm only planning on doing read-caches. Write caches have the nasty problem of unattributed writes due to the way the page cache works -- I guess I'll hold off on that for now. I really wanted to integrate with fscache, but its still not in the kernel and I think the patch may end up being too invasive to do speculatively.

I guess the other thing I may try to squeeze into 2.6.21 is the shared memory transport (talking to Inferno primarily, but I'll probably also do mmap for rHype, and I'll look into architected shared memory with Xen). Next step after that will be looking at XIP (execute-in-place) modes so that I can start looking at a shared page cache. I really have to look into how this would work, probably won't make it into 2.6.21.

I'd still like to see Security, Async I/O, common id mapping, and a rework of the extended modes -- but none of those things are central to my primary work on 9p, so they'll have to remain on the back-burner for now.

cachefs->9p->vacfs->venti on Linux

2007-02-09T09:04:00.001-06:00

I've been thinking a lot about dealing with file systems on clusters using venti. Part of this departs from the centralized file server model of Plan 9 to a system more resembling the Linux kernel source control topology -- which has a centralized repository and then a hierarchy of distributed repositories "cloned" from the central, each having private branches/sandboxes.

In its most simple form, the idea would be roughly the same, the central file server (for the cluster or whatever) would have a venti repo. Indiividual machines would connect to the venti server and store local caches of whatever branch they've attached to. Any mount of the central file server must be associated with a particular version which root's the client's branch. Updates to the central server or the branch are explicit, not implicit. So we replace replica/pull with a venti/pull which remounts the file system with a newer score.

The venti servers can be hierarchical as well, with explicit pull/pushes from eachother. Data sharing is accomplished by sharing scores and merging. So we keep centralized authority, but also have the authority for relatively independent subdomains.

There are several things that might be nice to see to help facilitate this (and maybe they are already there)-

better method for non-local venti access -- allow venti communication through the file system versus just TCP. This is a trivial bit of shell scripting I suppose, but it would be nice to be able to apply user/group permissions to venti channel access
establish a meta-data tree in venti that can be referenced without knowing its score. This meta-data tree acts as a table of contents with pointers to published branches/scores, etc. Use standard file permissions and whatnot as rudimentary control. This could be used to implement git-style "porcelin" allowing venti as a version-control-system with tags, log messages, etc.
git-style three-way merge tools that help merge trees. Venti has always been more of a historical archive than concurrent versioning system -- adding three-way merge utils would allow us to broaden its scope and functionality.

The eventual goal is to replace fossil as our primary root within Plan 9 (and also bring it to Inferno) - but as an intermediate goal, the idea would be to use cachefs (or cache-files I can never remember which is which) under Linux to provide a cache for a 9P-mounted vac file system and then work out some scripts to help with the push/pull'ing of stuff and maintaining the meta-tree.

A side-project would be to look at porting the git porcelin to work with venti as a back-end and port the porcelin to Plan 9/Inferno so we have a version control strategy that works well with our native content-addressable-strorage.

PS3

2007-01-07T11:06:00.001-06:00

Got the PS3 from FedEx two days ago. I'm on the third day of downloading the Linux DVD. Then I'll get Inferno hosted working, followed by taking cracks at Inferno native and Plan 9 native.

Browser as GUI

2006-12-19T14:02:00.000-06:00

The various projects where I am working with Inferno and Plan 9 at IBM use the technology "behind the scenes" -- that is to say, it does not focus on using either as the Desktop or user interface. In my opinion, we as a community spend far too much time focusing on getting a browser to Plan 9 or creating a more traditional desktop environment for Plan 9. None of this plays to the strengths of the operating system.

I've been toying with the heretical idea of turning things the other way around. Instead of bringing the Browser to Plan 9, bring Plan 9 to the Browser. For instance - how much could be done with only presenting user-interface via a browser interface? This builds upon the idea of the Venti web interface (where you can get status from Venti via http) -- the idea is to leverage more advanced browser capabilities (AJAX, etc.) to provide the foundation of the UI. This certainly is less flexible than a draw based interface, but would likely provide a decent foundation for most status/control applications.

The work would be on the back-end -- developing an API which would create a draw-style synthetic file system representing the DOM of the front-end interface as well as files representing the aggregate interfaces to be presented by the web-server.

Over the next few months I plan to play around with developing such an infrastructure with a limited scope in support of my projects.

technorati tags:Plan9

Blogged with Flock

Extensions

2006-12-19T09:53:00.000-06:00

I have been doing some re-thinking on extensions. Right now we have extensions in the create operation and a bunch of extensions in the stat structure. This has been judged undesirable by the community at large - more of the proposals for extensions revolves around using an alternate aname mount for various extensions.

The more I think about it, the more I think what we really want is "meta-walks" -- that is to say "special" walks off the fid of a file or directory to get at extended information (which is then read/write using normal operations not stat/twstat). For example walk("/etc/motd/lock") to get to lock controls or walk("/etc/motd/xstat") to get at an extended (or just UNIX) stat structure.

The one thing that doesn't hold up under this is special files. Hard links, symbolic links and device files all "want" to be created atomically w/their extended attributes in tact. It would also be nice to be able to mark which files have "extended" attributes so we don't need to do multiple walks of a file to determine if its a symlink. I suppose we could use a DM bit that is just DMSPECIAL which could be reserved for just such a purpose. But that doesn't help with create -- I think the only way around it is to have a "special" create operation which is part of the extension definition. This fits in with Russ and Charles' "acceptable" extensions methodology.

So, we remove all current extensions -- the extensions for extended stat information, error strings, etc. will be moved to meta-walk items. Extended create will get its own operation.

The real question is the difficulty of an implementation. One nice thing is that it seems quite likely that this could be implemented as a "filter" file system over a "normal" file server (such as fossil) - squireling the extended data away in "invisible files" or perhaps an alternate hierarchy -- but then again, I suppose that would always be true as long as you had your own exportfs.

If we didn't have the special open -- we could use exportfs as is -- the key is the atomicity problem. I suppose we could create the file as a DMTMP and/or ORCLOSE-- after we write the extended information, we twstat it to DMSPECIAL. If something happens during creation (disconnected server), the file removes itself. I need to think about this more to make sure it would work and not create any turds or race conditions. Also need to make sure I can do meta-walks without someone getting in the way and protect meta-fids from race conditions including the removal of their parent file.

Inferno Shared Memory Channel

2006-12-19T08:19:00.000-06:00

Added a "generic" version of my Inferno shared memory channel transport which works using standard UNIX shared memory. The code is available on the txinferno branch of my git repository on git.9grid.us. Essentially, it looks just like a devip style /net device now with pluggable transports. Right now the only one is usm (unix shared memory), but I'll be adding the Xen and PAPR version shortly.

Listen semantics aren't quite right since every conversation represents a point-to-point connection, but what I have works for things like Styxlisten. Need to go back and add some control messages to allow modification of default buffer size, polling intervals, etc.

I suppose at some point I'll add support to v9fs for using such a transport as well, which would allow for really fast Linux<->Inferno communication on the same host. Although I'm not quite sure what I'd use that for right now anyways.

Xen Woes...take 3.0.3

2006-12-13T21:59:00.000-06:00

Richard Miller's success stories of Plan 9 and Xen inspired me to give it another go. Since I felt like vegging and watching TV tonight anyways, i decided to reinstall my laptop with Xen + Ubuntu Edgy + Plan 9. Trying the Xen demo CDs or their Express CD's led to utter failure. I think went and tried to install Edgy, and then use Synaptic to grab Xen. This mostly worked (at least for Dom0), but it still isn't finding my Atheros network card driver (even though I installed the restricted driver set for the Xen kernel). I suppose it should work fine as long as I have it plugged in, but its frustrating..too frustrating for this late at night.

technorati tags:Xen

Blogged with Flock

GIT repositories

2006-11-28T08:17:00.000-06:00

I've setup a git repository on http://git.9grid.us/git

Right now, the only important thing there are my PowerPC-Linux Inferno modifications, but I intend to keep all my public modifications/extensions to both Plan 9 and Inferno there.

Infernal Musings

2006-11-28T08:06:00.000-06:00

Recently, I've picked up Inferno as a 9p server and resource gateway for some work projects. In some ways it makes me question the whole npfs/spfs path. Hosted Inferno provides a stable 9p gateway to local UNIX file systems, with a working multi-threaded model and authentication.

The only real problem is that its a 100 pound gorilla for file serving. If I want to advocate people use Inferno for a server-side for v9fs, I don't want them to have to download 12MB of fonts or any of the other stuff that they won't need.

Similar to plan9ports, I sorta wish Inferno was nicely split up into packages so I could download the core, and then download whatever packages I want/need. Both Inferno and Plan9ports should look more like Gnome or KDE from a packaging perspective -- they are environments, but I need not download everything to use some part of the environment.

Additionally -- it'd be really nice if Inferno had a mode where instead of functioning as a virtual OS in hosted mode -- it operated more "in the native environment" -- which is to say, I'd like to be able to run a program.dis from my normal UNIX shell and have it "do the right thing":
* be able to handle a #!/usr/inferno/Linux/386/bin/emu prefix so I can use binfmt-misc
* use the root of my UNIX name space for the Inferno root namespace
* use the UNIX current-working-directory
* use my UNIX home dir instead of having to have an Infernal homedir
* have a "side-namespace" for Inferno synthetics and paths (not sure if this is the right model)
* ultimately be able to support graphics apps with X11 as the window-manager and support resize

This should allow us greater reach for Inferno within hosted environments and would make it a more palatable alternative as a server environment.

Perhaps I'll toy with these ideas on the plan enroute to Spain.

9grid.us Diagnosis: Not Good

2005-11-13T19:28:00.000-06:00

Finally got some time to look at texas.9grid.us -> looks like his drive is fried. I can try a re-install or perhaps move it to a different machine -- the development model CompactPCI chasis I have it mounted in probably isn't the best thing in the world heat wise.

I have a couple of replacement drives I can try. I could move to CompactFlash - but then it would be mainly read-only affair (ie. no hosting a mirror site or anybody's private user directories). For now, we are in limbo (which is why you've been re-directed here). I'll update this site with status, hopefully I'll have something together before December.

Purpose

2005-03-17T17:18:00.000-06:00

Setting up this blog to post about ongoing work on Plan 9 and related technologies.