Fedora 32 mini-review on the Blackbird and Talos II

It took a little time due to hardware hijinx and the present chaos at the day job, but as is traditional, here is the regular mini-review on the latest release of Fedora (now F32) on OpenPOWER. Fedora is what we run here at Chez Floodgap/Talospace, largely because when I got my first Talos II in 2018 Fedora 28 was the only distro that worked right out of the box. However, it's also advantageous because it incorporates updates at a much faster pace than other distributions (which is why you could boot a T2 with it back then), and it's instructive because problems found here can be sorted out hopefully before they get to folks on more conservative distros. That's why I think it's worth all of us caring about it, even if you don't run it. This post is on Firefox 76 on F32.

With this release Fedora 30 progresses to end-of-life, which it hit two days ago. F31 will remain supported until F33 comes out in 6-ish months, when it too will have roughly a month's grace period before being EOLed. As before, I update Fedora manually rather than through GNOME Software. If you haven't seen the process in awhile, it looks like this:

sudo dnf upgrade --refresh # upgrade DNF
sudo dnf install dnf-plugin-system-upgrade # install upgrade plugin if not already done
sudo dnf system-upgrade download --refresh --releasever=32 # download F32 packages
sudo dnf system-upgrade reboot # reboot into upgrader

This process did not go quite as smoothly as before; two packages (that apparently were dependencies of other things) were unupgradable, and the usual --skip-broken couldn't clear the logjam, so I had to dnf remove then by hand. In addition, there is an annoying bug that started sometime in F31 where reboots seem to end up using the serial port as the console no matter what the setting is in Petitboot (you have to do a shutdown and power on instead). Fortunately, I fixed TenFourFox FPR23 to finally work properly with the new web-based BMC firmware, so I could watch the action from the Power Mac G5 using the OpenBMC's serial-to-LAN redirector (see screenshot). If you are on the console, however, for some reason the upgrader on both my Blackbird (with a naked BMC) and my T2 (with a WX7100 workstation card and the firmware in BOOTKERNFW) still reboot up to a black screen. Switch to another VTY, like CTRL-ALT-F2, and log in as root, and you can periodically issue

dnf system-upgrade log --number=-1

to watch the action. The install took a little less than an hour. It automatically restarts afterwards.

Relatively little seems to have changed in terms of system performance (on kernel 5.6.x). VLC seemed a bit quicker on my no-GPU Blackbird at pushing pixels, though mplayer is still the speed champion for media playback. Running Wayland on the Blackbird was yet again an unmitigated disaster and I have not seen any information to suggest this has changed for simple framebuffers in general (so if you're a straight BMC-only system as mine is, fuggedaboutit). On the T2, Wayland is still not as snappy as Xorg, but performance was not massively different and window theming and decorations seem more consistent. However, several games still won't launch in Wayland such as DXX, and because I'm dependent on appmodmap and there is still no equivalent feature in GNOME or Wayland to find out which window is active, Wayland on the whole remains a loss of functionality over Xorg for little gain in my book. startx forever.

If you use a lot of GNOME extensions like I do, then be prepared for some additional fallout as usual. Argos, which I use to power my little DIY IPMI fan monitor, doesn't appear to be working right in GNOME 3.36: scripts update the menu bar but their actual drop-down menus don't work, so I'll have to look into it since the maintainer isn't anymore (it looks like someone else is working on the issue, and this diff seems to fix it). Dash-to-Dock would not update properly with the GNOME tool in Firefox for a couple tries and then suddenly started working for reasons I still don't understand.

However, the worst issue was GNOME crashing every time I opened the Applications drawer. Nothing showed up under All, and if I clicked on Frequent, GNOME would either reset itself or actually force me to log out. After some poring through journalctl -e, I realized it was Appfolders Management that was the problem. Disabling that got me the apps back, and it's not really necessary anymore either since you can drag and drop apps into folders without it now. Other than modest UI changes, though, I don't notice much different about the current GNOME except that my build of Firefox Nightly now has a weirdly elongated icon in the top bar.

Overall, F32 is not a compelling release, but after a couple false starts it's working, and once I got past the usual growing pains there have been no serious problems. I'm hopeful that IBM will take a bit of a firmer hand with Red Hat and especially Fedora in the near future: I'd like OpenPOWER to finally graduate from the "alternative architectures" and be a first-class citizen with x86, which I also think will bring more attention to the port, and hopefully its niggling polish issues can be dealt with. But Wayland is still horrible without a GPU (and isn't feature-comparable with Xorg even with one), and GNOME continues to wreck its own extension ecosystem with wild abandon. Red Hat has substantial influence on these projects as many of these projects' developers work for them. Maybe we should be paying more attention there as well.

Give your OpenPOWER machine a big K1ss

This may be the wrong message in this era of COVID-19, but another operating system option for your POWER9 is to give it a big kiss. That's Kiss Linux, by the way.

Kiss Linux is an independent musl-based non-systemd distribution advertising itself as having "a focus on simplicity and the concept of less is more." Notably it uses busybox as both a source for core utilities and as init, and packages build from source a la Gentoo.

Ostensibly x86_64 only, there is an unofficial ppc64le port available specifically advertised as compatible with the Talos II and Blackbird family. A Debian netboot version is offered which can run directly from Petitboot as a staging area to do the full manual installation with a pre-compiled tarball. Although I don't see any reason why it couldn't be made to work on POWER8, the tarball is compiled for POWER9, so pre-ISA 3.0 systems will need to do more work and there doesn't seem to be current support for big-endian.

On the package side, most things seem to work. Firefox is advertised as functional, which is a good sign because that also means all of its prerequisites (rust, build system, font shaping, etc.) must also be. Netsurf is also advertised, my favourite little browser that can. However, Chromium and LibreOffice are missing, and since Kiss Linux seems to lack dbus this would also exclude GNOME. But, hey, there's no Wayland either, which is good news if you're running a system on just the BMC framebuffer.

If you're running or experimenting with giving your OpenPOWER machine a big sloppy kiss, wipe it off first (ewww), and then post in the comments.

When will OpenPOWER OpenBSD be now? Soon.

The tease of the week is this tweet from the OpenBSD maintainers, which indicates ppc64 (as "powerpc64" and implying big-endian) support is coming to OpenBSD.

It should be noted that this initial patch is very preliminary, largely just the rudiments for getting the kernel to boot, but that's more than it's done previously. It should also be noted that locore.S has dependencies on OPAL (i.e., the Open Power Abstraction Layer), which is provided by Skiboot, so pre-OpenPOWER systems like the Power Mac G5 need not apply (the G5 in particular is better served by OpenBSD/macppc). However, I don't see anything in this first pass that wouldn't work on POWER8.

When I get my "old" dual-4 Talos II up and running again this sounds like something worth experimenting with, and this would probably be the easiest route to getting NetBSD up on OpenPOWER hardware as well (my personal BSD of choice which I run on several systems currently, including a Macintosh IIci). Meanwhile, if you're going to go full Dark Helmet and want a BSD on your OpenPOWER systems that's available Now, look at FreeBSD, which is currently the most mature BSD available for our machines.

The case of the disappearing core

Here's a fun pro-tip: what do you do when one of your system's cores went out to lunch and never came back? On my original dual-4 Talos II my compile times got abnormally long and more sluggish. In dmesg I noted with alarm that it was reporting numa: Node 0 CPUs: 4-15 instead of starting at CPU 0. That means an entire core (because they're SMT-4) somehow went off-line! What gives?

The answer turns out to be related to Hostboot. The GUARD portion of the PNOR controls what hardware components have been disabled (which includes RAM sticks and individual cores), presumably due to defect, but it can also happen spuriously if Hostboot mistakes a driver glitch for actual hardware failure and erroneously turns off that component in the hardware guard entries. With main power off, a simple pflash -P GUARD -c at the BMC root prompt will clear the guard entries and indeed the prodigal core returned forthwith when I powered it back on again. Thanks to Tim Pearson at Raptor for the #protip.

#ShowUsYourTalos

Today's #ShowUsYourTalos is a Blackbird, but hey, we're all family here. Brad describes his unit thusly: "POWER9 DD2.3, 8-core; 64 GB RAM; Fractal Design Meshify C case; be quiet! Shadow Wings 2 140mm front fan; Fractal Design Dynamic X2 120mm rear fan; Silicon Power 512GB PCIe 3 NVMe 1.3 SSD; Samsung Evo 850 1TB SSD (behind motherboard); Radeon Pro WX3200 GPU; FreeBSD 13-CURRENT." Nice! We like seeing some BSD action!

If you have an OpenPOWER system you'd like to show off, post in the comments or E-mail me.

Firefox 76 on POWER

Firefox 76 is released. Besides other CSS, HTML and developer features, it refines that somewhat obnoxious zooming bar a bit, improves Picture-in-Picture further (great for livestreams: using it a lot for church), and most notably adds critical alerts for website breaches and improved password security (both generating good secure passwords and notifying you when a password used on one or other sites may have been stolen). The .mozconfigs are unchanged from Firefox 67, which is good news, because we've been stable without changing build options for quite a while at this point and we might be able to start investigating why some build options fail which should function. In particular, PGO and LTO would be nice to get working.

QEMU adds POWER10 support

"Wait," you say, "POWER10 isn't out yet!" No, it's not; even Axone (variously known as the AIO POWER9 "Advanced I/O" and "POWER9 Prime") is still yet to come out later this year (for that matter, we haven't heard much more about the LaGrange-based Raptor Condor which was announced at the same time). However, there are almost certainly simulations of the forthcoming 7nm POWER10 in IBM's chip labs, and some of that is public in the QEMU sources.

To be sure, what's currently in QEMU is at most rudimentary. The patches that have landed in the new 5.0 release describe the POWER10 as "very similar" to POWER9. The DD1.0 initial stepping (PVR base 0x00800000 as opposed to POWER9 0x004E0000) apparently introduces new ISA 3.10 (POWER9 is 3.0), though the MMU and CPU initialization code for POWER10 right now looks like a copy-pasta from the POWER9 section and doesn't expose any obvious new registers. Similarly, I can't find any IBM documentation on 3.10, so we can presume that's either under wraps or yet to be completed. However, a related commit in Skiboot drops the codename for the POWER10 machine under development: much as the POWER8 prototype was Palmetto and the POWER9 prototype was Witherspoon, POWER10's prototype is named Rainier (presumably for the mountain and not the Simpsons character). Given Axone's release expected later in 2020, the first POWER10 system descended from Rainier (like the AC922 was descended from Witherspoon) will likely not appear until 2021.