International Workshop on Plan9 - live streaming video powered by Livestream

International Workshop on Plan9 - live streaming video powered by Livestream

Grande

As a quick experiment the other day, I wrote a file-system back-end for Venti which serves the Venti protocol, but instead of using Indexes/Arenas/etc. for the back-end, it creates a hierarchical directory structure similar to Git with a file per score. This was done in the context of some other experiments (which I will post about later when they are complete), but I figured this may useful in of itself. The code is currently available in my vdiskfs branch of Plan 9 Ports on bitbucket (http://bitbucket.org/ericvh/vdiskfs/src/tip/src/cmd/venti/grande.c). It is very much a prototype, so not very robust -- but feel free to use it or build on it if you think it is useful to you.

acme-js

I wonder how hard it would be to craft a web 2.0 ACME -- the basic widget sets are all there more or less, the new portal examples from the prototype JS libraries would be most of the work, just need handles to allow stretching the portals and a bit more interactivity in the title bars. Chording may be a bit of an issue, but according to the W3C spec I should be able to detect events for all three buttons.

Of course, there is more to it than just the interface. The back-end would be a file system providing both the traditional ACME synthetic file server and a separate directory (or perhaps view) with the necessary UI components (as JS and whatever). I was thinking a third view might be necessary for interaction with other users, but perhaps that could all be managed from the system view.

stats

For profiling and regression tests it would be nice to log snapshots of certain system performance counters (/dev/sysstat, /dev/swap, etc.) before and after the kernel of the benchmark (which may not be the entire application). It would also be nice to be able to enable periodic logging (although that would interfere with many benchmarks -- particularly OS noise). Periodic logging interference could be mitigated by placing the log process on a different core than the application kernel core (when possible).

I think the general idea would be to have a file in the namespace which can be written to to either snapshot the stats or start periodic logging on a particular core. The file could then be read to get the snapshot data in some convienient format. If this was all done on a single open that may even allow for several simultaneous users -- although the specific scenario that we are using probably wouldn't require that.

Tree and Torus Woes

Some of my modifications to cpu (specifically chdiring after backmounting the initiator to allow for a poor-man's desktop extension model) seems to have tickled a big bug on the CPU nodes -- namely the torus starts getting errors even though the torus isn't being used (by anything). Could be a sign of major memory corruption, but I doubt the cpu application itself could be the cause. More likely the backmount which becomes the CWD of the compute node starts issuing lots of 9P requests over the tree and things fall down. Of course it could just be a slightly different binary tickling the system in a slightly different way and producing a corner-case error condition that scrambles memory. At least it seems to be reproducable.

CPU, Desktop Extensions, and Caches

Okay - having the default be mounting our home directory seems to be a really bad idea. ftq- which writes one line at a time takes f-o-r-e-v-e-r to complete just writing out the data. We need a custom cfs that does aggressive cache and also does some write-back caching as well for this to even remotely work.

Of course that will fly in the face of doing this from all cpu nodes to a single file.

This turns out to be a nice driver, doesn't it? So what we want is a single cache running on the IO node and the cpu nodes mounting that. That could be a matter of switching to the chaining cpu method (of course that would require more modifications to cpu.c).

The other thing I've been toying with is the idea of being able to pass a setup script or namespace script to cpu in the same manner than xcpu allows. This would allow us to establish a default namespace, including mounting relevant file servers and/or setting up cache servers.

The other side of this is that this work will all be completely obviated once we have the execution model in place -- but it is useful to have this naive approach as a point of comparison to justify the execution model.

HARE FastOS Workshop Talk Video Available

The video podcast of my talk on HARE for the FastOS June 2009 Workshop is now available for download or from iTunes. Slides are available separately here.

On regressions and progress

Initial take will just use the single threaded ftq for regression, we still need a regression script to organize results, but this should be relatively straightforward since all the version info is availabel in the #P device. Not clear how we get md5sum of the binary unfortunately, we can make certain assumptions based on surveyor. To keep things simple, we'll start with hardcoded version and then generalize.

bugs in lat_ctx may be timing related, potentially complicated by the virtual machine. Need to get criswell rebooted and try out there to make sure its not something stupid tripping things up. Be good to have ftq and 9bench numbers for a typical x86 machine to calibrate against anyways.

Nope FP errors happen on criswell as well, something is buggered in the timing code when the measurements are too small.

Okay, not a whole lot of progress today. Futz'd a bit with the lmbench code to see what works and what doesn't. It will be a nice set of numbers at the end of the day and we have the x86 code to compare against.

Since criswell is running the same hardware as the box next to it I should be able to get numbers for Plan 9 vesus Linux easily enough and we can use these as baselines for the normal Plan 9 regressions. We'll need to get the kittyhawk and/or zeptos code base running and try either lmbench or our modified versions there.

Of course, the thing I was suppose to be doing today was getting a top-level script together which runs on the desktop and initiates data collection (initially only of ftq) and stores results in a hierarchy based on kernel configuration (as read by '#P'/version and md5sum 9bgp.elf). I suppose I may still be able to pull those basics together this evening.

Regressions

Its become clear in the process of the HARE work that we really need a regular regression suite to verify functionality as well as track performance over the course of development. It strikes me that whatever we come up with may be generally useful for the Plan 9 kernel since it seems we don't really have any quantitative tracking of kernel performance.

Seems like one of the first things we need is better information about the kernel encoded in the binary. We've got the date in KERNDATE of when it was build, but little information about where and when it was built (and also, by who).  This seems increasingly important in an environment where we aren't just building out of a single directory. For my purposes, some sort of tree identifier such as a venti score or a git HEAD would be useful to encode to track exactly which version of the kernel we are testing. Eventually it seems like it would be a good idea to have a unique id for the build tools and root file system as well.

Regressions will be stored in a temporal directory hierarcy (similar to yesterday), then a directory with the $OBJTYPE filled a directory labeled by the md5sum of the kernel binary at the leaf. This directory will contain an info file with extracted meta-data about the kernel and a numbered subdirectory per run collected. Each of the numbered subdirectories will contain a console output file, and a test output file per test (named by the test). There will also be an info file with any relevant information about the run (in the case of BG/P this will be the personality and bump-id).

The regression binaries will be included with the kernel as part of the initial ramdisk to make things simple at first. The script will be modular and run from the front-end node, requiring nodes be able to boot to the point where one could cpu into them.
This will also require us to be able to determine the assigned IP address of the IO Node(s) automatically. It makes sense to go for 128-node configurations with 2 IO nodes so we can test the Ethernet bandwidth between IO nodes and not just to the front end.

The idea right now is to have a set of 5 initial performance regressions covering computation, sparse memory access, OS interference, file system access, and network access using a typical 4-way SMP configuration. Eventually we may broaden these tests to cover a range of configurations including uniprocessor and large-page. The general idea is to have a short regression which runs around a minute or so for simple sanity check of the system at boot-up and have a longer regression which runs on a daily schedule and does a more comprehensive evaluation.

For the longer test one option might be to measure the time it takes to build the entire system from scratch (kernel build or kernel+applications) on every Blue Gene node hitting a single file server for source files and storing results to a ramdisk. A variation on this would be to develop a parallel build system in which all 128 nodes collaborated on compiling the system.

Another idea I am considering is the use of linuxemu to run Linux benchmarks under Plan 9 to both measure the kernel and the emulation infrastructure. Candidates would include lmbench, postmark, dbench, and netperf.

The initial focus is on gathering results, but the idea is one we have started collecting data to allow web-based visualization of the results tracked over time so we can monitor the general health and performance of the system, As use cases materialize for testing we'll add them to the matrix.

FastOS Podcast Launched

I went through and worked out the creation of a video podcast including setting up the right iTunes syndication. I'll be posting all the FastOS talks (including the HARE/Plan 9 one) eventually. They (and the slides) are available on the FastOS site (http://www.fastos2.org) and via iTunes

Its fairly easy, assuming you have a good mic for the speakers. I may try to go and clean up the old IWP9 talks we have video of, but as a community it would really be a good idea to start making video podcasts of talks, tutorials, and whatnot. All the kids are doing it these days after all. I'm thinking of going back and capturing all of my past talks that I still have the slides for.

Plan 4?

Perhaps instead of changing names or increasing numbers with Plan 9, we should reduce numbers in subsequent versions since we strive to have less code and complexity (unlike certain other operating systems).

Of course, there is the problem that all the other Plans did fail to enslave humanity. Still, there would be something nice to a Plan 4 which was more compact and simple, and it could use a reduced operating protocol 4P for communication. Of course, counting down from 9, we have a short window of opportunity to succeed.

Inferno ACME BG/P Environment

Have everything setup now to be able to launch and mount a remote inferno and then interact with the host using its devcmd and file system (and its network).

This lets me deal with BG/P (and cobalt and telneting to the nodes) completely inside ACME -- except the Inferno telnet (inside ACME anyways) is spewing CRs and not echoing my input.  Blech.

ACME Trivia

So Feedkey allows me to setup ACME to prompt for Factotum challenges -- but how do I get Feedkey to start every time I start ACME?  It doesn't show up in the dump, so its not as clear as autostarting other aspects of the system.

export over ssh stdio

So, I've modified the lcmd stuff a bit and am able to mount a remote inferno instance that I start with an os ssh:

(local script)

#!/dis/sh

load std expr string

args := $*

s := sh -c ${quote $"args}

mkdir -p /tmp/lpipe

bind '#|' /tmp/lpipe

echo os

os ssh $remote /home/ericvh/inferno/Linux/power/bin/emu -I -r /home/ericvh/inferno /dis/sshexport /tmp/lpipe/data1 &

echo shell

mount -A /tmp/lpipe/data /n/remote

/dis/sh

echo halt > /n/remote/dev/sysctl

unmount /n/remote

(remote script)

#!/dis/sh

load std

or {ftest -e /net/cs} {ndb/cs}

bind -c '#U*' /n/local

bind -a '#C' /

bind '#|' /tmp/pipes

cat /tmp/pipes/data > /dev/hoststdout&

cat /dev/hoststdin > /tmp/pipes/data&

export / <>/tmp/pipes/data1 >[2] /dev/null

echo halt > /dev/sysctl

The problem is that if I terminate the local inferno, the remote emu still runs. &nbsp;I'm not sure how to catch and clean that up gracefully....

converting push to pull

Forsyth pointed me at the Owen papers with the labor exchange methodology for grid workload management. It works more on a pull principal (as does Cilk workload-stealing come to think of it).

The basic model behind PUSH's distributed pipelines are that the workers "pull" their data from the pipes -- but the problem is that the work is currently uniformly distribtued by the fan-out process. What we need is the equivilent of an alt which will distribute new work to workers with the capacity for it versus getting jammed out round-robin'ing work.

Call for Interns

I'm starting the selection process for a 6 month intern position working on Plan 9 technology for the DOE project (details at http://www.research.ibm.com/hare). The 6 months would have to be between Sept of this year and August of next. Graduate students or post graduates preferred. You'll need a strong background in C programming with Plan 9 and/or Inferno experience preferred. If interested, I need a resume and availability by this time next week (June 2, 2009) and we'll need to chat on the phone by the end of next week.

There will be a second opportunity for a 3 month position for a separate (but still somewhat related) project - I'll post more details on that project when it becomes available -- it may be during next summer (2010).

Please send resumes, recommendation letters, and availability to me (ericvh AT gmail DOT com).

lcmd games and 9P tunneling over ssh

So I've started playing around with lcmd (see ipn lab 83 for more details) -- and its got the right data path for what I want, but I need a slightly different application than simple rcmd.

Effectively I want to use os from one Inferno to ssh to another machine and then use the ssh pipe to allow the two Inferno instances to mount eachother. The will allow me to securely traverse an ssh-only firewall and get more or less full service connectivity between the two inferno's (so that one can use the others network, or file system, or underlying host, etc.)

First stage is to use the ssh tunnel to either import or export a file system - I suppose import makes the most sense initially. The second stage is to build a 9P mux which differentiates which pipe to service depending on the 9P message header (directing responses to a client pipe and requests to a server pipe).

Simple experiments with lcmd seem to fail under OSX -- but I was trying to do them between acme-sac and emu, so its possible that one or both have issues with their hoststdio implementation (of course it could be something else I'm doing wrong as well) -- I do notice slight differences in behavior with hoststdin between the two -- acme-sac seems to report all activity, while regular emu seems to be muxing between a process reading hoststdin and another thread (perhaps the shell).

Come to think of it, the ipn stuff does an enormous amount of work to mux the hoststdin and hoststdout into a single pipe, why not just do this from the get-go? &nbsp;This applies to both hoststdio as well as the os command stdio -- easy enough to fix in both cases (I think).

In the case of the OS command, its just in how its written. &nbsp;The underlying devcmd just differentiates at the op level, not at the open level. &nbsp;So we just need to modify a line in os.b to use the same descriptor for both stdin and stdout. The hostio isn't quite as straightforward due to some hard-coded permissions. &nbsp;In order to preserve existing semantics, i'm just going to create a new Qid for r/w io.

The other problem here is that the target system is actually linux-powerpc. An open question is whether to port acme-sac there just to keep the source base consistent between client and server, or whether to attempt to use traditional emu as the server (implying changes in both).

acme sac does seem much better under OSX

Wow - I guess I had never really tried ACME-sac out on OSX. It seems much better integrated. &nbsp;The mouse scrolling works (which doesn't sound like much of anything, but really makes a difference), the default fonts better match the Mac. Its making me reconsider some of the client-side configurations I was considering for a new working environment for dealing with Blue Gene.

So -- what would be the implications of going with ACME-sac for the front-end instead of a vanilla Inferno? There was some consideration of using Inferno as a drawterm replacement, but that sort of faded due to latency concerns. 9cpu seems to be present, but broken -- perhaps its something that is easily fixed.

EDIT: Of course trying to fix things ends up revealing how much more restricted of an environment it is. I was never a huge fan of the Inferno wm-based debugger, but it did make problem determination significantly easier than without it. The case could easily be made that the right thing to do is "port" some of these wm applications to acme -- the common difficulty is one of time to do these sort of things.

So, what's the more pain in the ass, back-porting improvements from acme-sac into inferno or porting over "missing" items to acme-sac? "If I were king", I suppose I would take a slightly different approach:
# kill the binaries in both targets to ease source control woes mentioned earlier
# structure a generic build script like p9p to allow easy builds without binaries and modifying mkconfig every, freaking, time
# create a distribution target that would build the acme-sac configuration directly out of inferno

Of course, I'm not in direct control of either of these distributions, but I think if I do branch Inferno for Brasil, maybe I'll follow this guidance myself.

Source Control Woes

Blech. I've been trying to work in a branch of my mirror of the inferno-os tree on google code. The problem is that it contains a bunch of binaries which royally screws with the source control system. &nbsp;I removed most of the binaries, but of course now I can't pull from that tree without it freaking out.

As far as I can see, I have two alternatives:&nbsp;

* meticulously create (and maintain) a gitignore file so that every time I rebuild it won't put binary changes into the commit. This will (probably) fix the merge issues, but I'm not certain.

* create a full fork of the inferno-os tree and manually patch source files based on the changesets from svn. &nbsp;Actually, it would probably make the most sense to just fork a brazil-os tree from the inferno-os tree and strip out absolutely everything we don't really need.

EDIT: I've gone with gitignore for now, but a bit unsure how to commit the file back to the central repository.

UEM: Incremental Approach

I realized I was getting a bit overwhelmed by the complexity of what I described in the whitepaper and it was leading me to write more and implement less. I'm going to switch gears and go for some small, incremental steps to get the ball rolling.

First things first, I need a method for initiating commands on Inferno hosts using only a synthetic file system. Noah has the basics for this within exportcmd, but I need to go over it and make sure it fits my goals.

Second, I need the ability to mount that file server on OSX, Linux, and Plan 9 -- preferably over the stdio of the Inferno
process. There's an Inferno Lab which covers something similar that I can use as a base.

Next, I'll need to be able remotely mount this service while simultaneously exporting my namespace to it by using a mux on the communication channel which appropriately directs requests.

Then I can get into extensions which are necessary for devcmd, but the more I think about it, the more I think many of these extensions may be better implemented from an application level versus trying to do them within the driver (mostly things to do with namespace manipulation).

Also (where possible), I'll need a mechanism within devcmd to actually back mount an Inferno sandbox where I can setup the namespace for the target application.

Once all of this is in place I can move on to thinking about the front-end file system which is used to reserve resources and allocate nodes. &nbsp;I'm leaving this for last because it will require a plugin module and I'm still not entirely sure what form that will take.