A quick trip to IBM OzLabs

(Before we begin: this post was not sponsored, vetted or in any other way written in official collabouration with IBM; I'm writing this up strictly as a Power ISA bigot enthusiast and for no other purpose or consideration.)

Jetlagged greetings from a lovely trip now back at Floodgap Orbiting HQ in sunny southern California (this post partially composed on Air New Zealand's in-flight WiFi). Many thanks to Hugh Blemings of OpenPOWER whom I met at SCaLE 17x and suggested, since I was going to be visiting family in Australia and having a holiday in Canberra, that I head by IBM OzLabs (where he used to be manager). With the kind indulgence of Leonard Low, the current manager, it actually came together and on a warm autumn day last week my wife and I trooped down the National Circuit to the office.

OzLabs has a very long history as a Linux hacker collective and was one of the first commercial labs set up for Linux and Linux software support. In fact, it is rather infrequently reported that a visit to the Canberra Linux User Group, an indirect ancestor of OzLabs, is where the Linux Tux mascot originated (from a 1994 incident at the National Zoo & Aquarium where Linus Torvalds was actually bitten by a penguin; a somewhat inaccurate sign at the Zoo commemorates the injury). Formed at what was then the Australian division of Linuxcare in 1999, for a period of time OzLabs even provided a Linux CD burning service (in exchange for cookies, which my wife calls biscuits) along with Linux software development and support until the division shut down. Hugh was manager for part of this period along with its resurrection under IBM in 2001 as part of the IBM Linux Technology Center, which Leonard manages now.

The current OzLabs location is in an executive building and isn't purpose-built, as Leonard pointed out somewhat apologetically, but still gets the job done. Today the division concentrates primarily on Linux on Power support (including Skiboot and Petitboot, which actually originated from the Cell-based PlayStation 3) in addition to its hosted projects; in fact, Paul Mackerras, the original maintainer of PowerPC Linux, is today a senior technical staff member working on KVMPPC and very politely reviewed my trivial patches for KVM-PR a few months back.

Leonard greeted us at the entry and took us back where the magic happens.

Now, this is IBM, so there's still the corporate face. (I have a story about this: when I was an AIX sysdweeb back in the antediluvian days, we would regularly get visits from IBM salesdroids. However, a few years ago when I tried to buy my own POWER7 hardware personally, I couldn't get any VARs to take my money probably because I didn't need a service contract. I ended up settling for a lightly used POWER6 from a reseller; that box still runs Floodgap today. Now that I have an executive position in a large municipal department, though in a job unrelated to computing, upon hearing this story the IBM salesdroid servicing the municipal account gave me his card and told me to call him any time.) There are meeting rooms and a decent-sized auditorium space, where my wife was talking academia shop with Leonard while I shutterbugged.

A Thomas Watson-esque THINK mural dominates the wall (I have a THINK notepad as a gift from that salesdroid which I use for late night call notes). There's also a small display case nearby with several items, most notably the head and disk assembly from an IBM 3380 circa 1980. This assembly is a Model J with two actuators each accessing about 630MB each; the 3380's frame carried two of these assemblies for a grand total of 2.52GB in the AJ4 configuration, making the 3380 the first gigabyte storage device. The larger but slower Model K had 1890MB per actuator for roughly 7.5GB in a fully loaded AK4 frame, weighing a foot-flattening 250kg and pulling 6.6kW of power. This assembly alone weighs 32kg, so hope you had your hernia belt on while installing it. Due to its incredible rotational inertia the spindle was stopped by the equivalent of an automotive disc brake.

One disappointment: no photography inside the work area. Although I actually do hold a US security clearance, export regulations are such that pictures taken inside are not allowed, so to ease Leonard's heartburn all the pictures you see in this blog post were taken outside in the public space and I took no pictures within the secure area. However, I tapped out some copious notes and those I'll share with you.

In the prototype area Leonard showed us examples of Romulus and Witherspoon. The Romulus development reference design you should know very well by now: the Talos II is strongly based on it, and apparently the OzLabs developers like the T2 so much they're ordering more as workstations. (Maybe that's why it's currently backordered at Raptor, grr.) Witherspoon is described in Skiboot documentation as "a POWER9 system with NVLink2 attached GPUs"; it is the direct ancestor of the Monza-based AC922 used as nodes in the Summit and Sierra supercomputers (more at the end). There was also a small system with an FPGA prototype BMC under testing. Amusingly, the prototype room also had some historical items, including otherwise nondescript tower systems based on the PowerPC 604, 750 (G3) and 405, none of them Power Macs, and some workstation hardware I recognized from the beige days.

In the server room a POWER9 Zaius (a/k/a Barreleye G2) system was sprawled out on a table. This is an OpenCompute device developed jointly by Google and Rackspace as a successor to the original POWER8-based Barreleye. Although just 1U tall, the system we saw was too wide for IBM's racks, though I did rather like the removable drive bay. It takes LaGrange CPUs (more on Monza and LaGrange in a second).

We also saw the POWER8 Palmetto and Stratton prototypes in the racks, each in this SilverStone ATX case. The Palmetto design emerged as the Tyan GN70-BP010, the first customer-available OpenPOWER system; Stratton became the S821LC, with its close relative Briggs (get it?) as the S822.

Although the pamphlet I stole it from is dreadfully out of date (2008), since I couldn't photograph it you can get a small idea of the lab from this page out of IBM Australia's then-official brochure (warning: large PDF; usual disclaimers apply). While only some of the staff were there due to the Easter holidays, which was poor planning on my part, a relaxed and skilled atmosphere in the relatively open floor plan was evident. We also spotted the continuous integration display (all builds green!) and a modified xkcd that said "Petitboot" instead.

Michael Neuling, another IBM staffer, kindly provided some public chip samples to photograph and we took them outside the secure area. One of them was this POWER8 wafer which I took from a couple different angles. The two-ply "white" strip is test logic; the dies between them have six cores on a 22nm process. The Turismo POWER8 has one of these and the Murano POWER8 has two.

The POWER9 Nimbus scale-out family in OpenPOWER systems (on top of the POWER8's wafer carrier for size comparison, more or less), as elegantly hand-lettered by Mikey. Sforza is the chip we know and love in the T2 I'm typing this into; as implemented in Romulus it provides the most similarity to existing commodity designs and prioritizes PCIe, offering the most of all three (48 PCIe 4.0 lanes). LaGrange and Monza are in the larger form factors with double the memory channels of Sforza, with LaGrange also offering the biggest XBus bandwidth between processor sockets (two lanes, twice that of Monza and Sforza) and Monza the greatest OpenCAPI/NVLink throughput. Knowing this, it makes sense why Rackspace and Google went with LaGrange for Zaius, but IBM used Monza for Witherspoon/AC922 where GPU attachment mattered more.

At one point Raptor made an off-hand mention of a future LaGrange system, but so far nothing more has been heard.

Finally, a couple more items: the Top 500 certificates for Summit at Oak Ridge National Laboratory, TN and Sierra at Lawrence Livermore National Laboratory, CA, currently ranked numbers 1 and 2 as of this writing. Summit has 4,608 nodes based on the later 6-GPU AC922, each with dual 22-core Monza POWER9 CPUs and six NVIDIA Tesla V100 GPUs on a Mellanox dual-rail EDR InfiniBand network; Sierra has 4,320 nodes of an earlier revision with the same CPUs and four GPUs. Last is one of the many patents from the team, this one from 2015 honouring Andrew Bentley, a senior technology architect.

Although I couldn't show you everything, we're still very grateful we could drop by and see how the magic gets into our wonderful OpenPOWER machines. Thanks in particular to Paul, Mikey and Leonard for tolerating our silly questions and disturbing their quiet workplace, to Hugh for getting this all in motion, and all of the OzLabs inhabitants. We had a lovely time!

On our way out admiring the view from the auditorium (left: St Andrews Presbyterian Church; right: Parliament House) for the rest of our vacation, disturbed only by some officious stuffed shirt from the Attorney General's office that didn't like us photographing in a public street.