How far we've come in Power ISA

We haven't done a lot of chip pr0n lately, so let's do a little as a fun aside. Recently I bought a naked 601 (in this case a 601+, the second revision) since I figured it would look nice in my computer room along with my POWER9 die, shown at right in their display boxes.

Just how does that old 601, the very first PowerPC, compare with the POWER9? With painstaking precision propping my Pixel 3 on the end of a cereal box for height and eyeballing the distances with a Home Depot yardstick, I lined up both the 601 and the POWER9 die at the same scale, shown here.

The 601+ at left was manufactured on a 0.5 micron CMOS process in 4 layers, measuring 74 square millimetres. This is a die shrink of the original, which was manufactured at 0.6 microns in 4 layers and measured 121 sq mm. Both the 601 and 601+ have a single 32-kilobyte unified L1 cache, which was dispensed with for the 603, and 2.8 million transistors. At 80MHz, the original 601 consumed 8 watts; the 601+ die shrink reduced it to 4 watts at 100MHz.

The POWER9 die at right, on the other hand, is rather larger at 693.37 sq mm. This is almost six times larger than the original 601 and 9.3 times larger than the 601+, but with multiple cores and L1, L2 and L3 cache all crammed onboard in 8 billion transistors — almost 3,000 times more. Shipping POWER9 chips are built on a 14nm FinFET on SOI process in 17 layers and consume between 90 and 190W depending on core count at clock speeds from 2.25GHz to 3.8GHz.

In this oblique view you can see the functional units a bit better. The amount of power in modern Power ISA chips would have been absolutely unthinkable in 1992, particularly at modern process sizes. I, for one, welcome our future POWER10 overlords.

Finally, I've been meaning to post this picture for awhile that Mikey Neuling sent, with Anton Blanchard and Hugh Blemings showing off Microwatt running on an FPGA board (and Hugh's well-traveled Blackbird demo board) at OpenPOWER Europe in October. Now that's a mighty fine logo. The person who came up with that should be congratulated.

If you're States-side, happy Thanksgiving, and if you're not, eat some turkey anyway because it's better than that burger you were going to have.

MIPS Open Initiative isn't anymore

Less than a year after it launched, the MIPS Open Initiative has apparently terminated. Despite the happy talk still up on the main site, and no sign of public notification, Hackster is reporting that Wave Computing has terminated the offering with an extract from a legal letter stating it became effective on November 14. The complete letter is posted here. Indeed, the new sign up page simply 404s, though as of this writing the GNU tools downloads are still up.

It's debatable how open MIPS Open Initiative really was, compared to, say, OpenPOWER (or RISC-V or OpenSPARC). MIPS Open had a more legally problematic licensing structure called "open use" which only allowed registered participants to develop under the MIPS Open Architecture 1.0 license, requiring prior approval and developers under the MOA-1 license "to provide upon request by MIPS or its authorized MIPS Open Verification Partner, information demonstrating that such implementation in its current format is a MIPS Open CERTIFIED Independent Core." No community governance existed and anecdotal reports circulated that gaining access to critical developer resources was arduous and bureaucratic. For OpenPOWER, however, all an implementor has to do is meet the specification and obey the trademark requirements. More importantly, now that OpenPOWER is part of the Linux Foundation IBM can't backtrack even if they wanted to (and it's pretty clear they're happy with the response to OpenPOWER so far). And the free VHDL implementation Microwatt is just a Github clone away.

Meanwhile, clearly Wave doesn't want people to realize what's happened, given how delayed the news has been and how little announcement they themselves have made. Even less clear is what prompted the withdrawal, but the most likely theory is Wave is planning to put MIPS up for sale. Funded as an AI startup and lacking customers, the last thing Wave's backers want is IP they can't unload for dollars, which may explain why Wave CEO Art Swift was pushed out after four months. The question is whether anyone is commercially interested anymore.

Bettering the BMC and bring-up

Pretty much every computer these days has a service processor of some sort for bringing up the system and the main CPUs, just sometimes under different names (the Intel Management Engine could be considered a form of one, albeit with a lot more black boxes tacked on). So do POWER9 systems from the smallest Blackbird to the biggest IBM E980, and for many of these systems that service processor is the BMC, or the baseboard management controller. (Systems capable of PowerVM use one or more PowerPC (405?)-based Flexible Service Processors, or FSPs, which are accessed over ASMI. This includes big Power boxes like the E980 and even some z/Machines but also many pre-OpenPOWER systems like the 8203-E4A POWER6 that runs Floodgap. Since this is not particularly relevant to OpenPOWER, I won't talk about it further in this article.)

BMCs are present on multiple systems from multiple manufacturers, including many POWER8s and, most relevant to us, the POWER9 Romulus reference platform on which the Raptor Talos II family (including the Blackbird) is based. Most of these have AST BMCs, typically either the ASpeed AST2400 or 2500; all shipping Raptor systems use the AST2500. This is an 800MHz ARM11 ARM926EJ CPU with a secondary 200MHz ColdFire core, a built-in 2D framebuffer, USB and dual GigE MACs. Among its many tasks it controls the power rails, provides firmware stored in the PNOR flash for the main system CPUs, provides services to other autonomous subsystems over IPMI, receives temperature data from the main system CPUs' On-Chip Controller (OCC) and manages fan and environmental controls during normal operation. During bringup it is the main processing element before the primary CPU(s) are enabled; it is what signals the Power cores to execute from their burned-in OTPROM which then transfers control to the self-boot engine (SBE) firmware SEEPROM.

Besides the fact that the BMC has overwhelming ability to affect virtually all system components, including while the system is running, the BMC also directly influences how quickly the system can be brought up. To improve the security and auditability of BMC-based systems, the BMC in just about every OpenPOWER machine runs OpenBMC (not to be confused with the identically-named and similarly functioning Facebook OpenBMC), a small open-source Linux distribution tailored to its unique tasks.

The open availability of both the OpenPOWER firmware and OpenBMC itself is what really makes our systems truly ours from the firmware up. You can download and audit these critical pieces, including Raptor's Talos OpenBMC, and you are encouraged (and actively supported) to build and install your own. The problem, however, is that OpenBMC is relatively slow to get the system going when power is applied and the machine can't be started until it does. On a server which is normally up this is generally unimportant; my POWER6 gets rebooted pretty much only when the backup power fails. Similarly, this T2 is usually running all the time. My Blackbird, on the other hand, boots when the projector is turned on and gets shutdown when I'm not in the home theatre room. With over two minutes to get from turning on the power strip to a Fedora login and almost a full minute of that just to get the ability to start main power, this is a major drag and harms the ability to dogfood POWER9 in smaller applications. There is also the small but non-zero risk that if a power failure occurs during access to the flash that it could brick the system. The longer the bring-up time, the longer that potential window of vulnerability.

Fortunately, it looks like further advancements are now finally making a dent in the BMC bring-up delay. Almost 25% of the boot time on a Witherspoon (AC922) system was shaved off by converting the mapper service from Python to C++, and further savings were realized with straightforward wins such as eliminating other older Python components and adjusting the priority of the system service. Another big winner was apparently moving to dbus-broker, which is D-Bus compatible but higher performance. With all of these the OpenBMC bring-up on their Witherspoon box has reduced substantially and the upcoming AST2600 is reportedly three to four times faster.

This is a nice improvement even if it's probably most of the low-hanging fruit, and the OpenBMC team should get a solid thumbs up for the work here. I look forward to this appearing in a future firmware update for the T2 family. However, OpenBMC start time is only just one (albeit significant) piece of the startup puzzle: once main power is on, from the time the Blackbird boot screen appears (i.e., IPLs Hostboot) through Skiboot to the Petitboot menu the delay is still a hair over one minute, which compared to other platforms still seems way too long. Much of the time spent seems to be in Hostboot before Skiboot even gets initialized, but even Skiboot adds some overhead. Again, if you're like me and this is your primary computer, you won't deal with this often. But there's lots of Blackbirds and T2 Lites out there which are sidecars and while this is an obvious first world problem it's still a useability penalty to be paid.

None of this is a crippling fault with the platform, but particularly for the workstation market many of us are in, it's suboptimal. Therefore, continued improvement in basics like these makes the liveability of OpenPOWER on the desktop even better than it already is. And these improvements in OpenBMC hopefully should be just the beginning.

Debian 10.2 available

Debian and Fedora (and their various downstreams and derivatives) are probably the top two Linux distributions on Raptor hardware, and now Debian is updated to 10.2. This is a maintenance release primarily addressing security updates and some critical fixes. The ppc64el images are already available for download.

Fedora 31 mini-review on the Blackbird and Talos II

As promised here's my periodic mini-review after upgrading both our Blackbird and Talos II systems to Fedora 31, the most current release, typed up in Firefox 70 running on Fedora 31 on my T2. Even though there are many of you who don't run Fedora on OpenPOWER, these reviews are still relevant because Red Hat does a lot of the work on the components you do use, and problems are likely to turn up here first. Much to my disappointment, one late breaking note is that 128-bit long double still isn't in Fedora ppc64le, and didn't make 31 either. I can't tell from the bug or the wiki page why the deadline keeps slipping.

I did the upgrade first on my home theater GPU-less 4-core Blackbird because it was already bitten by the librsvg2 issue and early reports indicate the updated LLVM-rustc pair in F31 fixed it. The steps are the same as I used for F29 and F30 except for changing the parameter to --releasever=31 (duh). A quick check demonstrated updating librsvg2 to the latest available for F30 didn't solve the problem, so I went on to downloading the packages for F31.

When I rebooted into the F31 installer, however, the projector freaked out and went into an endless loop of trying and failing to sync to the display. I don't know if it was unhappy with the video mode the installer set, but even the A/V receiver wouldn't pass through the HDMI video (the T2 did something similar which I'll note in a moment). I eventually had to pull up a second VTY and then and only then would the projector display anything. I then logged in as root and monitored the messages from dnf with periodic dnf system-upgrade log --number=-1 | tail -10 until the machine rebooted on its own.

Fortunately, F31 came right back up. I've done only minimal customization on the Blackbird, so pretty much everything transferred over unchanged, and no packages had to be dropped to do the installation. F31 comes with GNOME 3.34, which is alleged to have performance improvements, and actually I was very pleasantly impressed as you can see from the screen shots:

Video playback on this GPU-less Blackbird was a lot better in this release; in fact, Firefox 70 didn't drop any frames or audio at all (though I'm sure the rapidly improving VMX/VSX support has something to do with it ;). Although VLC on the unaccelerated Blackbird is still not perfect and playback was not completely smooth, pixel pushing was much improved in both DVD and Blu-ray playback and there were fewer dropouts with the TOSLINK surround sound (mplayer of course still played everything just fine). Unfortunately, I think the improvements are strictly in Mutter and GNOME itself, not llvmpipe, because Xonotic was still only ekeing out a bare 5fps at 1920x1080 as in F30.

As advertised, librsvg2 was working again. If you have exclude=librsvg2 in /etc/dnf/dnf.conf, you should remove it before you do the update. Every GNOME release has some vanity changes for no good reason and the new icons and minor UI tweaks seemed largely unnecessary but they weren't objectionable. On the apps side, GNOME Videos doesn't seem to grok the length of my AIFF music files correctly, though it does play them (MP3 was fine). GNOME Web was also working again after a long hiatus but it seemed to have minor glitches, and since there are people who use POWER9 now who are actually helping to maintain Firefox, you should just use Firefox.

Since I didn't find any obvious major regressions in my normal usage, the next step was to update the Talos II. The T2 does not have the WX7100 firmware in the BMC PNOR, so I expect to run the installer "blind," but interestingly my LCD would not sync to the display either just like the projector wouldn't. The LCD synced fine when I popped open a VTY, just as with the Blackbird, so I'm thinking there's something up with the installer's video mode. Otherwise, the install proceeded unattended and rebooted uneventfully.

As my daily driver the T2 is rather more customized than the Blackbird. It's pretty much a given that I'll lose some of my GNOME extensions in the upgrade or the custom "classic" OS X-like theme I use will have some odd breaking edge case, and that happened here as usual. In this case Dash to Dock was the casualty and the GNOME Extensions Manager refused to update it, requiring me to manually install it. Tweaks and Settings still have visual issues with my theme, but didn't seem worse, just annoying. The only thing installed that didn't transfer over were my custom Perl libraries which got eaten and needed to be reinstalled. I know, I know, I'm the last person on Earth who still likes Perl apparently.

On the T2 with its WX7100 workstation card, the graphical performance improvements were not as notable as with the Blackbird, but some things seemed better, and a few 3D games that chugged a bit on F30 seemed faster on F31. I'd still say performance was a net win, just smaller.

Both systems use X.Org, but I do try to at least test Wayland. My T2 is configured to come up in a text boot so that I have a console to fall back on, an artifact of originally being Fedora Server and converted to Workstation. I was able to start GNOME in Wayland from the command line with XDG_SESSION_TYPE=wayland exec dbus-run-session gnome-session (instead of startx). I would call it incrementally improved from before. Some apps (mostly games) still don't start, and some games that do start have odd aspect ratios, but more at least work. The issue with some apps, particularly XWayland ones, not obeying GNOME theming seems to be fixed, and while it didn't feel quite as snappy as X.Org it was still better than previous releases. However, my custom appmodmap tool for dynamically remapping the keyboard only works with things that run in XWayland, because it watches X events to know which window is up, and the GNOME Wayland compositor currently has no plans to offer this information. So back to X.Org.

However, the situation was even worse with the Blackbird. Since overall graphical performance seemed better I decided to push my luck and see how it worked in Wayland (which previously ran like treacle on Thorazine in January north of the Arctic Circle), but as soon as I switched to Wayland and rebooted, this time the Blackbird would not come up in a graphical boot at all. On several test boots after the kernel messages it immediately went to a grey screen with the mouse pointer and then froze hard, requiring me to power cycle it because I couldn't open any VTYs or get to the OS. Since this is a workstation installation rather than a server installation converted into workstation, I had to boot the Fedora rescue installer to fix /etc/gdm/custom.conf because I couldn't get the machine to come up otherwise. If you are installing F31 from scratch, you may want to make sure that WaylandEnable=false is uncommented before you try your installation out.

Overall Fedora 31 is both a good release and a bad omen. Performance (at least in GNOME under X.Org) is overall much improved, especially if you don't have a GPU, but it's still obviously better even if you do. Some bugs were fixed and packages installed uneventfully. There were the regular growing pains in GNOME, but I didn't lose anything irreplaceable, and other than the usual bumps one experiences with custom themes and extensions pretty much everything just worked.

But Wayland on ppc64le continues to be worrisome. I must concede that at least on the T2+WX7100 things has improved since F30, and since I freely admit I'm a Wayland sceptic those of you who are heavily invested in it probably don't care about my opinion. But overall it's still a step backwards because there are still things that won't run in it, a big part of my own personal workflow may never work with it, and on the GPU-less Blackbird beforehand I couldn't use it and now I can't even start the machine in it. Meanwhile, Red Hat's made some very public signals that Wayland is the future and X.Org will be going away. In their rush to do so not much attention is being paid to people using 2D framebuffers with Wayland, and this is a real problem because no currently available GPU is libre and not supporting the built-in BMC in every shipping Raptor system is a waste (not to mention requiring people to incur additional expense just to get something to work that was "already working"). If you want a truly blob-free system, right now you just plain can't use Wayland, and it doesn't seem like they care.

DD2.3 POWER9 steppings now available

Raptor now has SKUs for the Sforza DD2.3 POWER9 chips, which they're calling "POWER9 v2". Currently just the 4-core and 8-core are available, but the higher core counts are presumably soon to come. There is a slight price premium of around 15-20% for these over the DD2.2 CPUs, but they fix a number of errata including functional hardware watchpoints (no more YOLO mode) and add the new Ultravisor mode for enhanced security (which will be the subject of a future article). In addition, although TDP, clock speed and cache specifications are the same, improved Spectre v2 mitigations in this stepping (specifically count cache flushing with hardware assist) mean possible performance improvements particularly for branch-heavy workloads. Support for this feature should already be in current Linux kernels.

If you have a T2 family system, you can order these today, and the SKUs are reported as in-stock. They are drop-in replacements for all T2s and Blackbirds and because their TDPs are the same can use the same heat sinks and HSFs. Systems shipping now may still have DD2.2 chips in them, though Raptor says you can get a DD2.3 for a slight upcharge.

Talos II and Talos II Lite officially FSF Respects Your Freedom products

No one disputes the Free Software Federation practices what they preach, and no one disputes that their standards are strict. So hats off to Raptor, who today officially received FSF Respects Your Freedom designations for both the Talos II and T2 Lite (here's the official announcement).

The designation recognizes that the T2 and T2 family have full system schematics and source code available for the entire firmware stack from the BMC up, and no keys are needed to update or replace any firmware component unless you require your own. (The same applies to the Blackbird, too, of course; presumably its own FSF RYF certification is soon to follow.) Naturally the designation presupposes you are using a free distribution, as the FSF defines it.

The T2 family joins a relatively small number of complete systems that have RYF endorsements and given those systems' loadouts is easily the most powerful, at least of this writing. Not only is this a nice win for Raptor, who have made libre computing a cornerstone of their company, but it's also a great validation for OpenPOWER. A designation like this from the FSF, who stakes their entire reputation on libre computing, is no small matter no matter how you slice it. Congratulations!

FreeBSD 12.1 available

FreeBSD 12.1 is now available. This is largely a maintenance release. To the best of my knowledge this is the BSD with the best track record on OpenPOWER so far; it is otherwise a relatively straightforward 64-bit big-endian Power implementation. I'm still a NetBSD dweeb personally (on mac68k, macppc, cobalt and hpcsh) and I'm looking forward to someone porting it sooner or later, but if you want a BSD on your Blackbird or Talos II right now this is probably your best bet.

The installation directions for the Blackbird should work as is for the Talos II. However, if you've already got the ISO (not the .img) dd'ed to a USB stick, it seems to me that it should "just work" in Petitboot without all the goofing around at the BMC prompt (if you don't, though, then these instructions will allow you to bring the machine "up from nothing").

If you are already running FreeBSD, unfortunately it does not seem that the PowerPC port of FreeBSD supports freebsd-update(8) yet, though I imagine this is planned. FreeBSD 13-CURRENT boots and runs fine on the Raptor family as well, but no clear word on when that will reach release yet.