Posts

Showing posts from 2022

Fedora 37 mini-review on the Blackbird and Talos II


It's been kind of a wild ride getting the Talos II and the Blackbird upgraded to Fedora 37, but we're there, so it's finally time for a mini-review to summarize the current state. As I always like to remind folks, Fedora was one of the first mainstream distributions to support POWER9 out of the box, it's still one of the top distributions OpenPOWER denizens use and its position closest to the bleeding, ragged edge is where we see problems emerge first and get fixed (hopefully) before they move further downstream. That's why it's worth caring about it even if you yourself don't run it.

Another important reminder is both my 'Bird and T2 are configured to come up in a text boot instead of gdm and I start GNOME (Blackbird) or KDE (T2) manually from there. I still test GNOME on both systems, but I've pretty much entirely migrated over to KDE Plasma on the T2, so I'll talk about both the GNOME and KDE experience in this and future mini-reviews. I strongly recommend a non-graphical boot as a recovery mechanism in case your graphics card gets whacked by something or other. On Fedora this is easily done by ensuring the symlink /etc/systemd/system/default.target points to /lib/systemd/system/multi-user.target.

As usual, the process is, from a root prompt:

dnf upgrade --refresh # upgrade prior system and DNF
dnf install dnf-plugin-system-upgrade # install upgrade plugin if not already done
dnf system-upgrade download --refresh --releasever=37 # download F37 packages
dnf system-upgrade reboot # reboot into upgrader

I did the Blackbird first, and immediately got a broken packages error because this was the system I had tested Pantheon on for Fedora 36. Just in case you didn't get the memo, new for F37 ain't no more Pantheon in Fedora (though there is a Copr). I just removed them instead (elementary-wallpapers-gnome, gala-libs and switchboard and switchboard-plug-tweaks). It booted into a graphical installer and rebooted without incident. The kernel release as of this writing is 6.0.13.

On the Talos II, which has the Raptor BTO option AMD WX7100 workstation card in it, even with GPU firmware loaded into the PNOR there was once again no graphical installation. If you manually select the kernel from Petitboot, it will at least show a text installation process. Alternatively, you can monitor on the serial port, or from a connected system viewing the serial console over the BMC's web server, or by logging into another VTY with CTRL-ALT-F2 or as appropriate as root and periodically issuing dnf system-upgrade log --number=-1 to watch log updates.

Although installation on the T2 also seemed to successfully terminate and reboot, Petitboot subsequently puked because it couldn't handle the state the updater left it in (the root is XFS and it had a stuck journal log entry). Fortunately the Blackbird was able to rectify the filesystem once the SSD was moved to an external device for recovery.

Both systems failed to update the grub2 configuration, requiring me to do so manually with grub2-mkconfig -o /boot/grub2/grub.cfg. This is a regression from bug 1921479 and can be detected in dnf with the same error message during a kernel package update (/etc/grub.d/10_linux: line XXX: test: XXXXXXX-pXXXXXXX: integer expression expected).

Additional problems were afoot on the Blackbird when starting a GUI, which is a basic 4-core using the ASPEED BMC for graphics. At least on the ASPEED, GNOME was rife with graphical abnormalities, worst of all in Wayland. (Starting Wayland from the command line also got more complex: I had to do something along the lines of XDG_SESSION_TYPE=wayland /usr/libexec/gnome-session-binary --builtin to get it to start; the regular gnome-session bombed out with an argument error.) Wayland is still restricted to 1024x768:

But that wasn't the worst of it. Trying to do even simple tasks yielded a lot of tearing and refresh problems. This is a picture of the screen because I couldn't even get the screenshot utility to work reliably.
So definitely a regression for Wayland. But even Xorg (still starts with startx, may need XDG_SESSION_TYPE=x11) had some unusual problems, like some apps' titlebars being transparent and causing graphical glitches:
Oddly, it wasn't all of them, though the Terminal was most affected. Newer or recently refreshed libadwaita-based apps seemed relatively immune, as did apps like Firefox that adhered to older GTK APIs. I keep the Blackbird as stock Fedora as possible, and I ran another update just prior to writing this up and didn't see any improvement.

KDE was not affected by that issue, though I observed — at least with Firefox and Thunderbird — that I had to grab the window and move it around a bit to get click point coordinates to be correctly reflected when the apps are full screen (then, after giving them a little shake by the titlebar, I could maximize them again and all would be well). I don't know whose bug this is exactly.

That particular irritation persisted on the Talos II, but none of GNOME's graphical problems that I saw on the Blackbird's BMC graphics. In fact, GNOME performed rather well for a change: I didn't need to force a rebuild of libgraphene this time to get improved performance and the UI was very smooth in Xorg. It also appears that the colour management issues I used to have where the screen would get blue-tinged have been rectified. Wayland GNOME had occasional animation stutters and a mild bit of lag but was otherwise useable, which is invariably the most I can say about Wayland.

Also in the positive category is that the bustage churn from the long double update in Fedora 36 is now almost all behind us, and the toolchain successfully built Firefox and other large projects without any new problems.

Overall F37 is a mixed update, more good than bad, but with some unwelcome regressions. In particular, if you are running BMC graphics only, even in Xorg GNOME has picked up some new glitches and in Wayland is once again a big mess. Systems with a GPU will largely be spared these issues — or just don't run GNOME. Likewise, be prepared with a second system to do any filesystem recovery if you're a long-time Fedora user and your root is still XFS; it may be time to convert it over to something else if you get pounded by it every time you do an upgrade.

Firefox 108 on POWER


Now that the Talos II is back in order and the Fedora 37 upgrade is largely behind me, it's now time to upgrade Firefox to version 108. There's some nice performance improvements here plus a hotkey for about:processes with Shift-Escape. Support for WebMIDI seems a little gratuitous, but what the hey (haven't tried it yet, the Macs mostly handle my music stuff), and there are also new CSS features. As before linking still requires Dan Horák's patch from bug 1775202 or the browser won't link on 64-bit Power ISA (alternatively put --disable-webrtc in your .mozconfig if you don't need WebRTC). Otherwise, we were able to eliminate one of our patches from the PGO-LTO diff, so use the new one for Firefox 108 and the .mozconfigs from Firefox 105.

When Petitboot barfs, everything's vomit


Colourful, no? But it's true. I've not been able to write up my Fedora 37 experience, nor upgrade Firefox (nor do further work on the JIT) because the Petitboot boot menu couldn't stop touching the main NVMe drive and making its older (Linux 5.5) XFS kernel module hang. If Petitboot can't start, your expensive POWER9 system is a brick.

In its most literal sense this article is largely a precautionary tale, because unless you're a long-term Fedora user like me with a continuously updated older installation, it's very unlikely you have an XFS volume in your OpenPOWER box. But if the antique kernel in Petitboot ever starts barfing on your own filesystems or a device you install, you'll be in this state too, so here's how I got the Talos II working again.

It's pretty much been a constant that you need a second system to deal with glitches. For me, this is usually my trusty Quad G5 Power Mac sitting next to the T2 which is connected to its serial port (or to the BMC's), and this works when it's a problem you can resolve from the BMC side, which is many of them. It would be nice to power up a Talos or Blackbird and have the console automatically start up talking to the BMC instead of needing another system to do so but this is what we have, at least until Kestrel develops that capability.

Unfortunately, this wasn't one of those problems:

  [Disk: nvme1n1p2 / 19a5d4e3-19f7-423f-a75b-5b15c8ee0bff]
    Fedora (0-rescue-ee275f6a7d994c9981e4e1436b83172d) 30 (Workstation Edition)
    Fedora Linux (5.18.13-200.fc36.ppc64le) 36 (Workstation Edition)
    Fedora Linux (5.18.18-200.fc36.ppc64le) 36 (Workstation Edition)
(*) Fedora Linux (6.0.12-200.fc36.ppc64le) 36 (Workstation Edition)
  System configuration
  System status log
  Language
  Rescan devices
  Retrieve config from URL
 *Exit to shell


 [fedora-root] Processing new Disk device[    8.041704] XFS: Assertion failed: !
(fields & XFS_ILOG_DFORK) ||
 (len == in_f->ilf_dsize), file: fs/xfs/xfs_log_recover.c, line: 3103
cpu 0x26: Vector: 700 (Program Check) at [c0002007e33171c0]
    pc: c008000008dc46bc: assfail+0x54/0x60 [xfs]
    lr: c008000008dc4694: assfail+0x2c/0x60 [xfs]
    sp: c0002007e3317450
   msr: 900000000282b033
  current = 0xc0002007e32c3180
  paca    = 0xc0002007ff7f5900   irqmask: 0x03   irq_happened: 0x01
    pid   = 649, comm = pb-discover
kernel BUG at fs/xfs/xfs_message.c:110!

After the assertion appeared, Petitboot locked up (at least on the regular console) and the system wouldn't start from any device because Petitboot could not be coerced into ignoring it. I tried holding down the x key from the serial console to force it into the shell, and that worked — but it still tried to mount the volume anyway and died. This did bring up a live kernel debugging session as you can see in the screenshot, but since I wasn't sure what the XFS module would do at this point and didn't want to risk the filesystem, I just powered it down.

Something about the state of the root XFS volume after the Fedora 37 update was making it go wrong, and I haven't been the first to observe this, either. Recovering cleanly would at minimum require a system that can mount and examine the XFS volume, and the G5, which runs Mac OS X Tiger, isn't that system. (Maybe the SGI Fuel next to it with IRIX 6.5.30 is — though that's something to explore some other time when it isn't my primary computer's boot volume at stake.)

Fortunately I've also got a Blackbird that did complete its F37 upgrade successfully. So it's time to do a little shopping.

I picked up two off-the-shelf NVMe-to-USB enclosures, one the Worst Best Buy Insignia store brand NS-PCNVMEHDE for about US$20, and a Sabrent EC-SNVE for about US$30 which also supports SATA. I was pretty sure the Sabrent would work due to their usual diligence about Linux, but I bought the Insignia anyway as a backstop in case the Sabrent was defective, and also because it came with a USB-C to USB-A converter since the Blackbird doesn't currently have any USB-C connectors.

Both devices are USB 3.2 Gen 2 and came up as "SuperSpeed USB" connected to the Blackbird's rear USB ports. The Sabrent is a much nicer unit with high-quality metal construction that folds open and has an integrated heat spreader in the top. The "tool free" part is there's a small clip that rotates to hold the M.2 stick in (with a stopper in the package for smaller-sized sticks). But even though the Insignia was kludgier (pulls out instead of folds open, requires you to stick on a heat spreader, really clumsy turn clip), it supports USB Attached SCSI Protocol; dmesg indicated the Sabrent didn't respond to a UAS probe. If I could have combined the chipset in the Insignia with the case of the Sabrent, we'd have the perfect enclosure.

Both devices also worked in Petitboot — by which I mean having the tainted NVMe SSD plugged in while Petitboot came up would also crash the Blackbird.

Bringing up Fedora first and then connecting the enclosure after, we next get the T2's root volume up so it can be checked. Because both the Blackbird's boot drive and the T2's boot drive have the volume group name fedora, we'll need to rename the T2's. We list the volume groups with vgdisplay; the T2's starts with lO, so the commands are:

vgrename `vgdisplay | grep lO | awk '{print $3}'` tfed
lvchange -ay /dev/tfed/root

But xfs_repair /dev/tfed/root wouldn't try to fix it: it said there was a log entry that had to be replayed first. This can be done simply by mounting it, so

mount /dev/tfed/root /mnt
umount /mnt
xfs_repair /dev/tfed/root

This showed no errors, so I inactivated the root LV again with lvchange -an /dev/tfed/root, disconnected the NVMe stick, put it back in its PCIe carrier and reinstalled it in the T2. Petitboot didn't crash, but Fedora requires the logical volume be named fedora, so we enter the Petitboot shell first and finish up with

vgrename `vgdisplay | grep lO | awk '{print $3}'` fedora

and then boot.

Whose bug was this? Well, arguably, Fedora might not have properly unmounted the drive after the update, but the error appears to be minor in that simply mounting the drive (with a later kernel, admittedly) fixed up the issue. It's more important that Petitboot have a stable, well-tested codebase, so the decision to use an older kernel (though 5.5 is a little excessive) is not an unreasonable one, and this older kernel appears not to be able to do that kind of recovery.

But if Petitboot can't do it, it shouldn't just brick the system. There should be a way for a user to hold down a key and bypass the menu without mounting anything, and try to recover in the shell at that point, which you can do from the console. Similarly, if it barfs on a filesystem or an installed device, it should simply say so and ignore it, not panic. These computers are just too expensive to have vomit everywhere when something goes wrong — and you shouldn't have to have a whole second system around to clean up the mess.

Linux 6.1


I'm a little behind on stuff since I'm waiting for parts to get my T2 booting again (doing everything on my Mac laptop and my long-suffering Quad G5), but kernel version 6.1 came out, and there's some really good stuff on Power to mention.

But first the marquee general improvements: first, general support for Rust in kernel, which is now fairly mature on Power ISA (every Firefox build I make has it) and has obvious security benefits — assuming you're on a platform it supports, that is. The other change I think is a big one, possibly even bigger than Rust support, is the enhanced multi-generational LRU (Least Recently Used) memory page evictor: it's not on by default, but it ships as a configurable option, and some of the reports show some impressive performance wins. Finally, the new implementation of in-kernel maple trees means better cache hit rates and less lock contention for those kernel structures reimplemented with them (if you're 64-bit and have an MMU, which naturally we do), and I know people will appreciate the updates to AMD GPU support.

However, the Power-specific improvements are particularly interesting. If you're using the POWER9-and-up radix MMU (not available on the POWER8, nor if you need to use KVM-PR thanks Russell Currey for the correction: HPT already has this support), there's now the option of execute-only mapping (as opposed to read-execute which is supported with hashed page tables). Another important Power improvement is full support for 64-bit Power ISA under both hashed and radix MMUs with KFENCE, a "low-overhead sampling-based memory safety error detector of heap use-after-free, invalid-free, and out-of-bounds access errors." Interestingly, 32-bit PowerPC was supported first!

To me, though, I'm most impressed with the exceptionally hard but worthy work done to rework system calls to use the new shared syscall wrapper implemented for s390, arm and x86 and obsolete the old legacy layer. This causes syscall handlers to take their parameters off the stack rather than relying on the state of the argument registers and r0, which is an obvious benefit if the registers are already on stack (such as for exception handling) because an additional stack frame wouldn't be needed, and further offers the opportunity to zero or sanitize them to prevent them from being used as a means to influence speculative execution (where expedient, and likely coming in 6.2). This has at most a minor performance boost, but it seems to be a definite security and maintainability gain, and best of all the new wrappers work on all PowerPC and Power ISA CPUs except the IBM Cell.

Expect to see it soon in Fedora and other leading-edge builds, and trickling down to other distros near you (full change list).

RHEL 9.1, AlmaLinux 9.1 and (eventually) RockyLinux


Red Hat Enterprise Linux 9.1 is out for those of you who pay money, but for those of you who don't, the new-CentOS-no-stream versions are coming up quick with AlmaLinux 9.1 now available for ppc64le and its other, less interesting architectures. As is typical for a point release RHEL 9.1 is largely a maintenance update, though it also includes support for Keylime, Red Hat's remote boot attestation solution. Keylime needs a TPM, though, which doesn't come with Raptor units as sold; there's nothing on the TPM headers on my three systems here. AlmaLinux's release notes are almost identical to RHEL's, including Keylime, and you can get ISOs from their mirrors or upgrade in place. RHEL customers will know their usual drill.

What about RockyLinux, the other white meat? 9.1 is still in beta as of this writing but is "passing ... to our testing team" for signoff. When ready, it should be ready for upgrades in place or new ISOs from the usual location.

As for CloudLinux, it seems to now be derived from AlmaLinux, so you might as well just use that.

Firefox 107 on POWER


Firefox 107 is out, a modest update, though there are some developer-facing changes. As before linking still requires Dan Horák's patch from bug 1775202 or the browser won't link on 64-bit Power ISA (alternatively put --disable-webrtc in your .mozconfig if you don't need WebRTC). Otherwise the build works with the .mozconfigs from Firefox 105 and the PGO-LTO patch from Firefox 101.

Fedora 37


Fedora 37 is out today. As I always say, it's usually one of the first mainstream distros to incorporate new changes and was one of the earliest distros to support POWER9 at all, so you should care about it because bugs and problems show up there first (if you don't like how the bleeding edge cuts your skin, try AlmaLinux or RockyLinux instead, which aim to occupy the niche old pre-Stream CentOS did). Chief amongst its changes is the new GNOME 43, which really does seem to have much better performance on OpenPOWER than previous releases (a big problem for the last couple) along with revised settings, toolkits and even more libadwaita-all-the-things, which means even fewer apps will respect your GTK theme. This also means Pantheon is no longer supported due to incompatibilities, so I guess I won't bother trying it again (admittedly it was definitely buggy even with GNOME the 42nd).

On the OpenPOWER side, the dust has largely settled from the 128-bit long double update, which was necessary pain and has translated into better package availability with the vast majority of regressions having been corrected. The kernel is up to 5.19, though 6.1 may arrive soon with possible good news for graphics card support on our systems. Some of us are tracking a problem with SATA PCIe cards, including ones Raptor ships as build-to-order options, using the Marvell 88SE9215/9235 SATA controllers (the earlier generation 88SE9128 seems unaffected), and I'll verify if this is still the case in the new kernel. The regression happened definitely by 5.15. gcc is up to 12.2.1.

Unfortunately, OpenPOWER is still in the AltArch penalty box (but along with aarch64 and s390x, so at least we have company), though that's better than ARMv7 (a.k.a. arm32, armhfp) which is no longer supported at all. I usually give it a week or two for any straggler packages to catch up and then I'll do our usual mini-review (here was the abbreviated one for F36). Note that I only test GNOME on my Blackbird; my daily driver Talos II is now KDE Plasma and not looking back. Should have made the jump ages ago.

POWER9 and tagged memory and why you care


Another excellent analysis by Hugo Landau (using findings from Jim Donoghue) on the presence — and accessibility — of hardware-supported tagged memory in the POWER9, even bare-metal POWER9s like ours. Operating systems like IBM i (formerly OS/400) use tagged pointers on every quadword for security purposes to mark pointers as valid, storing the tag data outside of the normal addressing space. If an invalid pointer is loaded, a trap instruction intercepts the fault. The instruction to set tags is undocumented and (apparently) privileged, and nothing other than IBM i currently uses it, but naturally that didn't stop these guys. Enabling tags active requires you set your POWER9 to big-endian and use the HPT MMU (i.e., the same configuration IBM i would run the CPU in). Hugo provides a detailed technical discussion on how they are accessed and stored, plus sample code (spoiler alert: the tag set instruction is 0x7c0103e6).

Firefox 106 on POWER


Firefox 106 is out, with PDF editing, the "Firefox View" feature for finding previous content on both your own desktop and any Firefox Sync-connected devices, and a big update to WebRTC. Of course, that only happens if you build with WebRTC on, and if you do you'll still need Dan Horák's patch from bug 1775202 or the browser won't link on 64-bit Power ISA (alternatively put --disable-webrtc in your .mozconfig if you don't need WebRTC). Otherwise the build works with the .mozconfigs from Firefox 105 and the PGO-LTO patch from Firefox 101.

OpenBSD 7.2


The latest release of OpenBSD is available, the new 7.2. Although there are few, if any, Power-specific changes, there are many welcome general ones including multiple improvements in SMP and updated graphics and hardware drivers. LibreSSL and OpenSSH are also updated. Note that the powerpc64 port remains big-endian only — which I have to admit is lovely — and the changelog page only gives "XXXX" for number of prebuilt packages available for 32-bit powerpc and 64-bit powerpc64. Here is a complete list of changes and download mirrors.

More cores for Mesa llvmpipe


On our open platforms we've long bemoaned that we currently need to deal with closed firmware to have good graphics performance (or indeed even 3D support: the built-in ASpeed BMC, though it has open firmware, only provides a 2D framebuffer).

While various alternatives like Libre-SoC continue development, the only 3D solution right now for a system that wants to run entirely open is a software rasterizer like llvmpipe, and even though it supports ppc64le its performance has not been great historically on our systems — see my poor struggling 4-core Blackbird running Xonotic at 1080p on the right. Fortunately, a modest but noticeable improvement is landing which should help. Apparently there's a hard cap of 16 threads, meaning all but the smallest 4-core Blackbird and T2 Lite machines were going underutilized, so now the cap is raised to 32.

This doesn't double graphics performance: as a developer notes in the thread, there are other bottlenecks that serialize the output, so the effective improvement going from 16 to 32 on a system with sufficient threadroom is about 10%. Also, on a smaller system the renderer will only use up to the maximum number of threads available no matter what the cap is set to. Still, if you have the cores this gets you another frame per second or two, so that's not nothing. Best guess is this will come out as part of Mesa 22.3; props to Luke Dashjr, who noticed the hard cap, and Jeremy Rand, who got the patch landed to raise it.

The next logical question is how far to turn up the volume knob (or, alternatively, why there's a hard cap at all apart from not recruiting too many execution units when the improvement is expected to be minor). While I can't answer the second question, Jeremy is looking for someone who wants to try ramping the patch up to 176 threads, the maximum number of hardware threads available on a dual-22 system. Such monsters do exist in the hands of enthusiasts, although it would also be good to see how it performs on smaller systems (regrettably my dual-8 is my daily driver or I'd try this already, so I'm deferring this until I have to mess with the guts again). If you're able to recompile your own local copy of Mesa with this change, post in the comments what you observe (there's a benchmark script on the Raptor wiki you can use to get the performance delta).

Linux 6.0


The Linux 6.0 kernel is out, formerly "5.20," and on its way to a distro near you. Keeping in mind that no Linux numerical release corresponds to any particular milestone, real or imagined, the marquee improvements include more graphics hardware support, XFS performance and scalability improvements (I like this in particular since my Fedora root is still XFS), further preparations for Compute Express Link, zero-copy send for networking and io_uring userspace block driver support.

On the PowerPC and Power ISA side this release is largely fixes, but notable new improvements are support for syscall stack randomization, a driver for the PowerVM Platform KeyStore, and atomic operations with the 32 and 64-bit BPF JITs. If you work on 64-bit Book3E hardware, now you also get full KASAN.

As a postscript in the no-country-for-old-hardware dept., support was withdrawn for the NEC VR4100 MIPS CPU family, which among other systems powers the IBM WorkPad z50 and the Agenda VR3 Linux PDA, and support for VMEbus was moved to staging with the threat to remove it entirely if there's no maintainer. Sadly won't be me since I don't have any VME hardware currently.

Tonight's game on OpenPOWER: ZGloom


We've done some DOS games recently, so for a change of pace let's do an Amiga one. Gloom was a pretty unabashed Doom knockoff using an Wolfenstein-type engine with some map geometry enhancements and transparency and palette effects. I actually have it on my A4000T and it's a bit blocky but plays pretty well with AGA and an '060. You know the drill: marine type guy shoots up other malicious marines, and, um, skinheads, and robots, and then ends up in a Gothic tomb as you do besieged by monsters and ghosts, and then goes to hell, takes the fight to the demons and kills a big dragon to save civilization. Just another day at the office! Its creators made it freely distributable and open-sourced the engine, allowing reimplementations to be made; probably the most developed of these is ZGloom. ZGloom isn't a perfect port — it's missing the title screens, the font and HUD are different, and there are various bugs ranging from trivial to moderate — but it has all the interstitials for what little story there is, has music and sound effects, plays well and does so at high resolution, and has configurable controls with X-axis mouselook. Also, because it's software-rendered, OpenPOWER systems with just the BMC framebuffer can play just fine (it has a multithreaded renderer to take advantage of all those shiny cores we've got). There's no save feature, but ZGloom simply gives you infinite lives, so just keep grinding away with impunity — and while you only have one weapon at a time, you can power it up, so grab all the bouncing orbs you can get for a real supercharge. Doors and switches are triggered by just walking into them, and baby bottles (!) give you health. There are other useful powerups you can find ...

I should also note for the squeamish that Gloom infamously came in "Messy" (temporary gibs) and "Meaty" (permanent gibs) modes, and this port seems to entirely play in "Meaty" mode, which means enemies explode and litter the ground with an alarmingly fast accumulation of body parts. On the original Amiga this would have brought lower-specced systems to their 16-bit knees, but this is a POWER9, so we can have all the fragmented torsos we want. (I have intentionally not shown this in the screenshot.) Don't say I didn't warn you.

Building it from source is straightforward; Fedora 36 has SDL2, SDL2_mixer and libxmp. Before you type make (or make -j24), however, edit the Makefile and add -O3 -mcpu=power9 to the CXXFLAGS. Then download this ZIP of the game resources, unzip it, and copy or symlink the ZGloom binary inside the resulting directory. While you can jump to any level from the main menu, game settings (graphics, keys, etc.) are controlled from the in-game menu after you actually start one.

Now things get serious!

Firefox 105 on POWER


Firefox 105 is out. No, it's not your imagination: I ended up skipping a couple versions. I wasn't able to build Firefox 103 because gcc 12 in Fedora 36 caused weird build failures until it was finally fixed; separately, building 104 and working more on the POWER9 JavaScript JIT got delayed because I'd finally had it with the performance issues and breakage in GNOME 42 and took a couple weeks renovating Plasma so I could be happy with my desktop environment again. Now I'm on track again with everything hopefully maintainable and my workflows properly restored, and we're back to the grind with both those concerns largely resolved.

Unfortunately, we have a couple new ones. Debug builds broke in Fx103 using our standard .mozconfig when mfbt/lz4/xxhash.h was upgraded, because we compile with -Og and it wants to compile its functions with static __inline__ __attribute__((always_inline, unused)). When gcc builds a deoptimized debugging build and fails to inline those functions, it throws a compilation error, and the build screeches to a halt. (This doesn't affect Fedora's build because they always build at a sufficient optimization level such that these functions do indeed get inlined.) After a little thinking, this is the new debug .mozconfig:

export CC=/usr/bin/gcc
export CXX=/usr/bin/g++

mk_add_options MOZ_MAKE_FLAGS="-j24" # or as you likez
ac_add_options --enable-application=browser
ac_add_options --enable-optimize="-Og -mcpu=power9 -fpermissive -DXXH_NO_INLINE_HINTS=1"
ac_add_options --enable-debug
ac_add_options --enable-linker=bfd
ac_add_options --without-wasm-sandboxed-libraries

export GN=/home/censored/bin/gn # if you haz
This builds, or at least compiles, but fails at linkage because of the second problem. This time, it's libwebrtc ... again. To glue the Google build system onto Mozilla's, there is a fragile and system-dependent permuting-processing step that again has broken and Mozilla would like a definitive fix. Until then, we're high and dry because the request is for the generated build file to be generated correctly rather than just patching the generated build file. That's a much bigger knot to unravel and building the gn tool it depends on used to be incredibly difficult (it's now much easier and I was able to upgrade, but all this has done is show me where the problem is and it's not a straightforward fix). If this is not repaired, then various screen capture components used by libwebrtc are not compiled, and linking will fail. Right now it looks like we're the only platform affected even though aarch64 has been busted by the same underlying issue in the past.

The easy choice, especially if you don't use WebRTC, is just add ac_add_options --disable-webrtc to your .mozconfig. I don't use WebRTC much and I'm pretty lazy so ordinarily I would go this route — except you, gentle reader, expect me to be able to tell you when Firefox compiles are breaking, so that brings us to the second option: Dan Horák's patch. This also works and is the version I'm typing into now. Expect you will have to carry this patch in your local tree for a couple versions until this gets dealt with.

Fortunately, the PGO-LTO patch for Firefox 101 still applies to Fx105, so you can still use that. While the optimized .mozconfig is unchanged, here it is for reference:

export CC=/usr/bin/gcc
export CXX=/usr/bin/g++

mk_add_options MOZ_MAKE_FLAGS="-j24" # or as you likez
ac_add_options --enable-application=browser
ac_add_options --enable-optimize="-O3 -mcpu=power9 -fpermissive"
ac_add_options --enable-release
ac_add_options --enable-linker=bfd
ac_add_options --enable-lto=full
ac_add_options --without-wasm-sandboxed-libraries
ac_add_options MOZ_PGO=1

export GN=/home/censored/bin/gn # if you haz
export RUSTC_OPT_LEVEL=2
I've got one other issue to settle, and then I hope to get back to porting the JavaScript and Wasm JIT to 102ESR. But my real life and my $DAYJOB interfere with my after-hours hacking, so contributors still solicited so our work can benefit the OpenPOWER community. When it's one person working on it, things are slower.

Void PPC goes Chimera (and bust)


Void PPC maintainer Daniel Kolesa has announced that instead of simply phasing out big-endian support in Void in 2022, he will instead cease maintaining the PowerPC/Power ISA fork of Void Linux entirely in favour of Chimera Linux, a fusion of a Linux kernel, musl libc and FreeBSD userland built with LLVM. There may even be a return of support for big-endian, at least for 64-bit Power (32-bit Power to be considered), as well as Chimera's core support for ppc64le, aarch64 and x86_64 (with 64-bit RISC-V coming).

As with the announcement for big-endian, if you like Void PPC and want to maintain it yourself, you can — provided you bring your own build and distribution infrastructure. Otherwise, Void PPC maintenance will end in January 2023. You can try Chimera Linux in the meantime and see how it works for you.

The RAD750's successor looks like it's RISC-V


It's been PowerPC in space for decades, from the Opportunity rover (a 20MHz BAE RAD6000, based on POWER1) to the James Webb Telescope (a 118MHz RAD750 "G3"). These are battle-tested processors in extremely hostile conditions, such as the Juno space probe in orbit around Jupiter where the RAD750 (a 132MHz part with 128MB of DRAM) operates in radiation levels a million times the human lethal dose. As evidence of its performance, it was supposed to be deorbited for destruction in 2021, but was extended to 2025 to examine the inner moons of Ganymede, Europa and Io.

Still, PowerPC never had a monopoly: the European Space Agency uses LEON, which is actually a free and open SPARC V8 core, and NASA has also used MIPS processors such as the 12MHz Mongoose-V (based on the R3000) used in New Horizons, which visited Pluto. A cluster of ARM-based "Rad-Tol Dependable Multiprocessors" (PDF) with OMAP 3503 cores will fly on the 6U CubeSat Lunar Flashlight nanosatellite scheduled for later this year after SLS Artemis 1 got scratched. For non-mission critical components, even some off-the-shelf ARM cores have made it to space; the Perseverance rover is another RAD750 system at 133MHz, but the Ingenuity helicopter drone it deployed was a regular Qualcomm Snapdragon 801.

BAE does have later generation Power parts available today: the RAD510 SOC, a system-on-a-chip with twice the performance of the RAD750, and the RAD5500 family with the RAD5545, derived from the ISA 2.06 NXP e5500. These are all Power ISA, all radiation-hardened, and all available from BAE's Manassas, Virginia facility, a U.S. Department of Defense Category 1A Microelectronics Trusted Source. (The RAD510 core is actually made by GlobalFoundries Fab 10 — one of IBM's former fabs.) With those on the shelf it's a bit puzzling that SiFive announced their X280 (U74-derived) core with vector extensions and AI/ML support will be the heart of NASA's next-generation High-Performance Spaceflight Computing (HPSC) processor instead, or more accurately eight of them (an additional four unspecified general-purpose RISC-V cores round out the total to 12). The chips are being developed on a radiation- and fault-tolerant process by Microchip Technology over the next three years, at a cost of US$50 million. More than just the added processing capability, probably what gives it a greater edge is lower expected power usage on a smaller process size and the ability to shut down silicon blocks for even greater power savings.

It would have to be indeed a significant technical leap to justify a complete break from a well-understood architecture and we'll see soon enough if it's worth it. That said, assuming it accumulates the same stellar track record as the BAE Power parts, the RISC-V HPSC will likely have its own decades-long run in space (assuming Jack Kang, SiFive's idiot senior vice president of business development, can stop smoking crack: there are still more PowerPC programmers than RISC-V programmers even now, chumpcakes). For that matter, the ESA is interested in RISC-V too and we approve of any free computing solution as long as it does the job. But don't cry for Power ISA in space yet; with three years of HPSC development to go, and several critical missions in progress, there's plenty of universe to explore no matter what CPU is doing the exploring. As HPSC will just be one choice of many even at NASA, Power ISA parts are likely to remain part of this very conservative industry for awhile, especially in commercial and military applications.

What's Arctic Tern good for, anyway?


Arctic Tern (and its associated soft-BMC Kestrel) is a product that's hard to describe just looking at it. Is it a boot accelerator? Is it a BMC replacement? Is it a development board? Is it OpenPOWER's answer to the Pi Nano? (Answers to pop quiz: yes (in the sense you get the BMC up quicker), yes, yes, and sort of but not really, since the clock speed is too low and it lacks some accoutrements.)

Raptor's new manual should address some of these concerns, and particularly covers the worry I and others had about bricking our system trying to get it installed (disclosure: yours truly reviewed a pre-release copy and submitted comments). Yes, it does need a PCIe slot if you want it to act as a VGA controller or USB host device; yes, it comes with all the necessary cables, including a JTAG programmer; yes, it's compatible with the Blackbird; no, soldering isn't required. But the instructions that are there are step by step and copiously decorated with illustrative photographs such that anyone reasonably handy with their machine should be able to do it.

Instead, the biggest roadblock to the casual interest will likely be the programming: you have to build the FPGA bitstream and firmware image yourself, and then flash your unit manually. The modules don't come preconfigured. (What, did you think that JTAG box was included just for giggles?) Raptor provides Kestrel source and instructions for Debian Linux on an OpenPOWER host in the manual, but the instructions are lengthy, and there is no public build or any other binaries presently available. This is primarily due to the rapid background development but also consider it a minimum "hacker" threshold required to join the club. Other operating systems on Raptor or OpenPOWER hosts should work, but x86_64 users will have to bring up their own toolchains. Raptor did say that at some point there will be "known good" images, though when that will be is an open question since the Kestrel source isn't even close to stable right now. Compared to that, the requirement that you also patch the on-board BMC to disable it in software seems comparatively minor (and it's reversible).

So who's this good for? Raptor may have their own ideas about that, but from over here typing this on my (now switched over to KDE) Talos II, right now I see this product really for someone who wants all the pieces in one place to play with Microwatt to develop their own embedded applications for it. If you're one of those people, you may not even care about the state of Kestrel anyway; to you it would be merely an example. On the other hand, for those of us looking forward to a fully auditable BMC, Kestrel seems to be working but it may not be sufficient (or sufficiently solid) to replace your BMC today and you'd have to jump through a lot of hoops to do it. If that's all you're interested in, you may want to wait. But it comes with everything you need to get started and for those in its apparent target audience, the barriers to entry don't seem unreasonable.

Linux 5.19


Linux 5.19 is released, largely with more hardware support and new architectures (including an unusual super-modern virtual Motorola 68000 platform to emulate Google Goldfish devices — my real Q800, clockchipped running A/UX, is flabbergasted). Here's a full list of changes.

This release doesn't have a great deal of Power ISA-specific improvements, but one generally positive note can be implied from Linus Torvalds' announcement message: he's now on an Apple silicon laptop, running Asahi Linux. Recall that Asahi Linux uses 16K pages as opposed to 4K pages, and this is still the case; many Power ISA workstation users are on systems that use 64K pages, including this Raptor Talos II running Fedora 36. If software bustage is happening for Linus himself, then it's a good sign that more software will start becoming more flexible about allowing larger memory pages and that's good news for us niche configurations too.

Arctic Tern available for purchase


Raptor now has Arctic Terns available for purchase, along with the user's guide. The US$1600 bundle comes with the PCIe carrier card, one ECP5 FPGA module (using Microwatt, the FPGA OpenPOWER core), FSI and JTAG adaptors and all the necessary cabling to connect to a Talos II system (we presume this means the entire T2 family including the T2 Lite and Blackbird, but it doesn't say — more about that in a moment UPDATE: it does now). As it is designed to completely replace the onboard ASPEED BMC, there are fan, Ethernet and two (!) HDMI connectors on board. There is a second module slot as we surmised, but it appears most board functions will be available with just one FPGA module installed, as provided in this bundle (fortunate since an extra module is US$900).

Unfortunately it looks like it does need its own PCIe slot and people like me with a nearly full loadout will be a bit disappointed if that's truly the case. We don't yet know, because the user's guide doesn't look like it has installation instructions for any T2 family system either even though it does have Raptor's usual studious pinouts and schematics. Being primarily a peripheral, I look forward to seeing additional documentation posted since no one wants to buy a $1600 card, get it home and accidentally brick it and their expensive OpenPOWER computer. Once I get my hands on one, we'll talk more about it.

Tonight's game on OpenPOWER: Duke Nukem II


No, not that Duke Nukem game — I mean the platformer. Before the Build engine wrought PG-13 destruction upon the City of the Angels, which also builds and runs on OpenPOWER, Apogee introduced the world's most egotistical alien exterminator in two episodes of heavily armed hopping around. The first installment in 1991 was poor even among PC games of the time, especially considering the far superior (and also Apogee-published) Commander Keen that came out the year before. But the second episode in 1993 had better graphics, better animation, better music, even a rip-roaring VGA cinematic if you had the hardware:
(Always wear your eyes and ears during target practice, kids!) It generally plays fine in DOSBox, but where's the fun in that? RigelEngine is a re-creation of DN2 that plays like the original DOS game mostly faithfully — pedantic quibbles shortly — along with various enhancements such as widescreen support, shown here in the screenshot.

RigelEngine builds out of the box on Fedora 35 and 36, though it has a rather surprising amount of vendored 3rd-party libraries and additionally requires OpenGL, SDL and SDL_mixer. Make sure to clone it with submodules enabled (e.g., git clone --recursive), then mkdir build ; cd build ; cmake .. ; make.

You'll also need a copy of the game, either the shareware first episode (1MB ZIP) or the full game (which I have, as an early DN3D purchaser). Apogee-3D Realms titles have moved from GOG, our usual drug dealer, to ZOOM Platform. Put the NUKEM2.* files into a directory and point the RigelEngine binary at it, or it will present a basic file picker when you start and then remember those settings.

As a clean room re-creation of the game, the additional features are simply incorporated into the game's regular settings menu (i.e., no specific command line options are used to enable them). Widescreen works just dandy with the exception of the radar and inventory frames which can sometimes blend in with the display a little too well; otherwise, I highly recommend it. On the other hand, the smooth scrolling feature — while being as smooth as advertised — makes playing the game feel a little like I've been stoned, uh, not that I would know anything about that, offisher (too used to those rapid 8-pixel and 4-pixel parallax moves). Also, while I'm being an ungrateful whiner, the introductory VGA cinematic is also not quite right compared to DN2 on my real Am5x86/133 DOS tower: there's an extra pause in the transition between "NEO LA: THE FUTURE" and Duke in the shooting range, and his firing rate in the first scene is too quick (it's fine when you're looking at the target). I know, I know, right? Uncanny valley!

Note that RigelEngine doesn't play the original Duke Nukem (this does), nor Cosmo's Adventure, which uses code descended but different from DN2 (this does).

This is too easy!

Raptor says the Blackbird crunch ends in August (and maybe Arctic Terns too)


Good news for everyone with a Blackbird backorder: Raptor is announcing order fulfillment and restocking by August 31, 2022. This may not mean the order you submit now will get fulfilled by then, but if you have your order already in, the wait will be over soon and new orders should be processed much more quickly. (This date does not apply yet to the Talos II Lite, but I'm sure Raptor is working on it.) In the meantime, if you can't wait, may we suggest a regular T2? Those are in stock and ready for purchase.

Raptor is also stating Arctic Tern will launch in the "next few weeks" for purchase, with the Kestrel soft-BMC onboard and compatible with the entire Raptor family including the full T2 and the 'Bird. We're looking forward to it and expect a review as soon as I can get my hands on a couple. Faster BMC booting is always welcome around here!

Rocky Linux 9.0


With new version 9.0 Rocky Linux joins the list of ppc64le-compatible CentOS clones, along with the already extant AlmaLinux 9 and Circle Linux based on Red Hat Enterprise Linux 9 (itself based on Fedora 34). Rocky Linux explicitly requires a POWER9 CPU. Other than that, the big difference is the branding and the governance, but more choice is always good. Download ISOs are available.

CXL is going to eat OMI's lunch


The question is whether that's a bad thing. And as it stands right now, maybe it's not.

High I/O throughput has historically been the shiny IBM dangled to keep people in the Power fold, and was a featured part of the POWER9 roadmap even though those parts never emerged. IBM's solution to the memory throughput problem was the Centaur buffer used in POWER8 and scale-up Cumulus POWER9 systems (as opposed to our scale-out Nimbus POWER9s, which use conventional DDR4 RAM and an on-chip controller), and then for Power10 the Open Memory Interface, or OMI, a subset of OpenCAPI. In these systems, the memory controller-buffer accepts high-level commands from the CPU(s), abstracting away the details of where the underlying physical memory actually is and reordering, fusing or splitting those requests as required. Notoriously, OMI has an on-board controller, and its firmware isn't open-source.

But why should the interconnect be special-purpose? Compute Express Link (CXL) defines three classes of protocol: CXL.io, an enhanced CPU-to-device interconnect based on PCIe 5.0 with enhancements; CXL.cache, allowing peripheral devices to coherently access CPU memory; and CXL.mem, an interface for low-latency access to both volatile and non-volatile memory. Both CXL.cache and CXL.mem are closely related and themselves transmit over a standard PCIe 5.0 PHY. Memory would be an instance of a CXL Type 3 device, implementing both the CXL.io and CXL.mem specifications (Type 1 devices implement CXL.io and CXL.cache, and rely on access to CPU memory; Type 2 devices implement all three protocols, such as GPUs or other types of accelerators). The memory topology is highly flexible. If this sounds familiar, you might be thinking of Gen-Z, which aimed for an open royalty-free "memory semantic" protocol; Gen-Z started the merge into the CXL Consortium, led by Intel, in January.

IBM was part of Gen-Z, but eventually let it dangle for OpenCAPI and OMI, and while it is a contributing member to CXL this seems to have been as a consequence of its earlier involvement with Gen-Z. But really, what's OMI's practical future anyway? So far we've seen exactly one chipset implementation from one vendor and that implementation has directly harmed Power10's wider adoption apart from IBM's own hardware. OMI promises 25Gbps per lane at a 5ns latency, but Samsung's new CXL memory module puts 512GB of DDR5 RAM on the bus at nearly 32Gbps. It's a cinch that Power11, whenever it gets on the roadmap, would support at least PCIe 5.0 or whatever it is by then and CXL would appear to be a better overlay on that baseline. Devices of all sorts could share a huge memory pool, even GPUs. Plus, a lot more companies are on board and that would mean a lot more choices and greater staying power, plus more likelihood of open driver support the more devices emerge.

There are still some aspects of CXL that aren't clear. Although it's advertised as an open industry standard, there's nothing saying it's royalty or patent-free (Gen-Z explicitly was, or at least the former), and the download for the specification has an access agreement. The open aspect may not be much better either: Samsung has an ASIC controller in their memory device but it still may need a blob to drive it, either internally or as part of CPU firmware (earlier prototypes used an FPGA), and nothing says that another manufacturer might not require it either.

Still, OMI has the growing stench of death around it, and it never got the ecosystem support IBM was hoping for; CXL currently looks like everything technologically OMI was to be and more, and at least so far not substantially worse from a policy perspective. Other than as a sop to their legacy customers, one may easily conclude there's no technological nor practical reason to keep OMI in future IBM processors. With nothing likely changing on the horizon for Power10's firmware, that may be cautiously good news for us for a future Power11 option.

Firefox 102 on POWER


Firefox 102 is out, not an earth shattering release but with some welcome privacy improvements. It builds out of the box on this Talos II using the PGO-LTO patch from Firefox 101 and the .mozconfigs from Firefox 95.

Firefox 102 is also the basis for the next Extended Support Release, with support for 91ESR (the current version) due to end on September 20 with the release of Firefox 105. Due to a family emergency, I've been out of the country for a bit and haven't been doing much with any projects, let alone the POWER9 JIT (this is why we need more people working on it!). Now that I've been back for a few days and more or less in the swing of things again, it's time to dust it off and forward port the current version to 102 so people doing ESR JIT builds for Fx91 can continue to do so with Fx102. I'll make an announcement and post a patch set when it's ready.

And now a real RISC-V laptop ... maybe


Phoronix is reporting the first production RISC-V laptop, (code?) named ROMA, with "a quad-core RISC-V CPU (although clock frequencies are not noted), a GPU/NPU accelerator [and reportedly other features], up to 16GB of LPDDR4/LPDDR4X RAM [and] up to 256GB of storage." This sounds great, except that I was seriously underwhelmed by the Allwinner D1 in the DevTerm R-01, so the lack of CPU specs is not encouraging. There are also two distinct process nodes for the System-on-Module, 12nm for Pro and 28nm for Normal, so there may be a wide gulf between configurations. On the other hand, it does prominently claim to be upgradable, possibly by swapping out the modules. Strangely, it advertises itself with an ARM SC300 secure enclave, which seems a bit odd as well.

The other thing that's not encouraging, which Phoronix correctly calls bulls**t on, is the proliferation of buzzwords (NFTs! Web3! AR! BINGO!) in the press release. You can register your interest and how many units you want, though I'm understandably not thrilled about signing up for a pre-order from an unknown potentially sketchy company. If actual product emerges, I'll try to get one, but right now this seems more like just another revolution of the RISC-V hype machine.

Fedora 36 mini-mini-review on the Blackbird


I said that Fedora 36 was when I was going to jump ship from GNOME, since I'm not happy with the Adwaita-or-nothing ultimatum GNOME 42 poses squarely at heavy themers like me. The problem is what I'm going to jump ship to.

For the past couple weeks I've been experimenting on the Blackbird to see what window managers and desktop environments seem to work well with Fedora on ppc64le before I try to migrate my main Talos II workstation to whatever I end up picking. But I also know a few of you are itching to upgrade and waiting to see if there were any problems, and of course for those of you running a distro other than Fedora, Fedora's going to find problems earliest. So, this will be a mini-mini-review instead of what we traditionally do: what I've been testing on the Blackbird and how well it appears to work, keeping in mind that my Blackbird is a GPU-less machine using only the ASPEED BMC for graphics and a 4-core CPU with 16GB of RAM. I'd call it "low end," at least within the spectrum of practical OpenPOWER desktops.

The upgrade itself went fairly smoothly. You know the steps to this dance by now, but if you're new to the club, here's the fancy footwork:

dnf upgrade --refresh # upgrade prior system and DNF
dnf install dnf-plugin-system-upgrade # install upgrade plugin if not already done
dnf system-upgrade download --refresh --releasever=36 # download F36 packages
dnf system-upgrade reboot # reboot into upgrader

There were no broken packages and no upgrade glitches, at least on the Blackbird (the Talos II has rather more packages installed). Actually, I was surprised, because this is the release that finally fixes 128-bit long double after literally actual years and watching the PowerPC tracker in Redhat Bugzilla has been very busy of late. But, at least for the vanilla install on this Blackbird, there were no problems. In fact, I know the packages available have increased by at least one, and that one's a biggie. More in a second.

The upgrade was on a black screen again, but if you select the kernel manually from Petitboot you should be able to follow along. Alternatively, you can monitor on the serial port, or from a connected system viewing the serial console over the BMC's web server, or by logging into another VTY with CTRL-ALT-F2 or whatev as root and periodically issuing dnf system-upgrade log --number=-1 to watch log updates. I left it to upgrade while I ate lunch and came back to a clean reboot and a Fedora 36 login prompt.

Recall also that I no longer run a display manager directly on either of my running workstations because of past irregularities with gdm (other display managers may work, haven't tried); I boot directly into a text prompt and do a startx from there. If you are running gdm, lightdm or others, you can probably just select the session type on login and avoid some screwing around with ~/.xinitrc. But first, let's get the Wayland Wasteland out of the, uh, way with a quick XDG_SESSION_TYPE=wayland gnome-session and a heavy dose of antiemetics.

I was unable to appreciate really any difference, good or bad, with Wayland in F36 compared to F35. It still won't support resolutions greater than 1024x768 on BMC graphics, and speed was neither better nor worse; llvmpipe-based applications in particular seem to run roughly about the same as previously, which is to say, not great. My verdict is it's not a regression, which is itself a small victory, and that's as much as I did with Wayland. I then brought GNOME 42 up in X11.
On purpose my Blackbird doesn't have any GNOME themes and only the default extensions that Fedora provides as standard, so this is as vanilla as it gets. Tweaks still allows my Mac pointing memory to deal with closing and window buttons on the left. With the usual modeline 1080p works fine over HDMI in X11.

Fedora has various spins and other window managers available. I'm not going to do everything on the list, but I did a few. The first one I started with is Pantheon, the GNOME derivative from Elementary OS, mostly because it seemed more Mac-like to me. It turns out that the current release is unusually difficult to launch without a desktop manager; even gnome-session --builtin --session=pantheon just puts up a blank background. This is the current launcher I put in ~/.xinitrc:

#!/bin/csh

gala-daemon &
io.elementary.greeter-compositor &
io.elementary.greeter &
(sleep 1 && echo '12' && sleep 1 && echo '25' && \
 sleep 1 && echo '37' && sleep 1 && echo '50' && \
 sleep 1 && echo '62' && sleep 1 && echo '75' && \
 sleep 1 && echo '87' && sleep 1 && echo '100') | \
 zenity --progress --title="Autologin in progress" --text="" --no-cancel --auto-close &
sleep 5 && killall io.elementary.greeter
io.elementary.files-daemon &
io.elementary.wingpanel &
plank &
io.elementary.terminal &
exec gala --replace

It's really ugly, but it does properly load the theming — unless I start up the greeter, the GTK themes seem not to stick. However, I have to kill it so it doesn't hang around. This doesn't fix everything because apps don't seem to update in the dock and a few core components still don't start up right, and you can't log out to the shell without opening a terminal window and doing kill -9 -1, but it's good enough to do this:

And it's also good enough to do this:
Yes, sports fans, the proof that the long double issue is fixed is that MAME is now buildable and now even a standard package on ppc64le. Good job, Dan and folks!

Overall I found the Elementary-on-Fedora experience ... plausible. There are still some pitfalls and I'm not sure how many of them are the fault of my own configuration or something specific to ppc64le or various high-level deficiencies in the packages, but at least one person says it worked great in F35, so maybe I just suck.

Moving along, our next port of call was KDE Plasma. This basically works.

You can enable this either with something like switchdesk, which is a little antiquated, or simply putting exec startplasma-x11 into ~/.xinitrc (or, I suppose, startplasma-wayland, or these alternatives). I was able to get a theme that was good enough working. I know of others using it with Fedora, and it's probably the best secondary and supported option other than GNOME.

I don't have any screenshots of Xfce in F36 yet, but I was experimenting with it on the Talos II for F35, so I have high confidence that continues to work as well. The same is likely true for LXDE.

I also did a couple oddball WMs just for fun and for some new ideas. Other than wmx, which was possibly a little too bare, surprisingly these are more functional than you might think. The first was WindowMaker.

If WindowMaker looks like NeXTSTEP, it's no coincidence since it's a deliberate recreation of the interface. I'm very used to it since my SAIC Galaxy 1100 (a HP PA-RISC 9000/712 "Gecko" workstation in a MIL-SPEC portable case) runs NeXTSTEP 3.3. It's also stonking fast and has little baloney compared to most other window managers, but the interface is not designed to be particularly configurable other than individual menu options and cosmetic themes, and it's not very Mac OS X-like (because Rhapsody, Mac OS X and macOS aren't very NeXT-like). There is a Fedora package for it. Put exec wmaker into ~/.xinitrc.

The other oddball window manager I've played with so far is amiwm. Yes, this is an Amiga Workbench clone. There is no Fedora package for it; I built it from the source code on the Lysator FTP site (Lysator brings back memories since NannyMUD was single-handedly responsible for lowering my undergraduate grade point average freshman and sophomore years), though there seem to be a few patched-up versions on Github. In operation it's pretty much exactly what it says on the tin, screens, gadgets, requesters, Workbench menus and all:

And the damn thing not only works, it's even swifter than WindowMaker and does have the close button on the left. Maybe I've found a modern Amiga I do want. Anyway, the upgrade seems to be good. Go to it.

Firefox 101 on POWER


Firefox 101 is out, mostly of interest to developers but with a few liveability improvements. I suspect breakage of our LTO-PGO patch will be a regular occurrence and Fx101 indeed broke the diff again, but after updating rust, cargo and cbindgen the optimized build works fine with this revised PGO-LTO patch and the same .mozconfigs from Firefox 95.

I can't figure out where the bugs are in our POWER9 Ion JavaScript JIT implementation, though I still strongly suspect it's because we use sign-extended 32-bit ints. This is enough to pass the test suite but still breaks with the web. MIPS does too and we are strongly based on the MIPS port, but one wonders if MIPS suffers from the same issues, and I'm quite sure it's not tested anywhere near as well. As 32-bit arithmetic instructions aren't orthogonal in Power ISA this change would require some significant rearchitecting and I've just come off a really bad week at work, so I'll just try to pull up the current JIT to 102 so that those of you building the new ESR can continue to use Baseline Interpreter and Baseline Compiler, and then it will be time for more major surgery on the source code. You can help (please!).

Mini-review: The Clockwork Pi DevTerm R-01, or RISC-V on the go


This blog is unambiguously pro-Power ISA, not least because I'm a long-time PowerPC bigot to start with, but also I think OpenPOWER — POWER9 specifically — is currently the best option for a practical yet truly open computing platform: competitive performance, auditable stack, solid hardware, and good and steadily improving software support, so there. And that remains my official editorial position as I type this on my trusty Talos II.

But that doesn't mean I'm not other-RISC-curious, and RISC-V gets a lot of ink these days. I'll say for the record I believe much of that ink is hype: RISC-V is only a performance threat on the low end, there's a lot of sizzle and little steak in present hardware choices, and while the ISA may be open the actual implementations vary greatly on that point. I've observed that there are two main markets for OpenPOWER workstations, namely people who want to support alternatives to the x86-ARM duopoly, and those who want a truly libre auditable platform (with some natural overlap between these groups). RISC-V can scratch the itch of the first group, but it's arguable whether the ecosystem does so collectively for the second. That said, all hype machines, to borrow a cow-orker's trenchant expression, easily transform into self-licking ice cream cones and that sort of salivary momentum is why RISC-V is here to stay.

One other thing that RISC-V and OpenPOWER have in common, besides a royalty-free ISA, is that workstations are neither architecture's core market. POWER9 (and Power10 even more so) is still primarily a server and big-iron chip, and extant RISC-V cores mostly lurk in embedded applications (especially the "too cheap for MIPS" segment). But Raptor workstations at least have nerd awareness, while only the even-less-frequently-encountered HiFive Unmatched meets the definition of a RISC-V PC, and the others are just glorified evaluation boards. And even some people complain about the price of that.

For me, though, it wasn't the price that was the problem (I mean, I've got two T2 systems and a Raptor Blackbird and I'm saving my pennies for an Arctic Tern, so I'm all in on POWER); it was the form factor. I'm out of KVM slots and I have too many boxes around the office. If I was going to play with RISC-V as a use-it-like-a-computer user, it needed to be something that I could set up to mess with and tear it down to recover the desk space. Why, it could even be portable. That would be nice. There's no portable OpenPOWER option (yet?), so if there's a totable libre system out there other than those old bizarro Loongsons I'd love to rock one.

So when ClockworkPi announced they were making a RISC-V spin of their DevTerm "portable terminal," and for just US$240 to boot, I said, "Take my money. No, seriously, take it." So they took it, and yesterday about two months later it arrived, fresh from off the DHL boat from COVID-infested Hong Kong Hangzhou Manifest Tech Co Ltd (wait, did I buy a car or a computer?). Today I'll tell you about it.

Here's what this review isn't: in general it's not a review of the CPi DevTerm itself, though necessarily I'll mention some things of relevance. There are many of these reviews based on its previous iteration using aarch64 CPUs (mostly Cortex-A53 and A72) and the R-01 is literally just a DevTerm with a core module swap. Everything you'd like or hate about the form factor largely applies to both flavours, so refer to any of those existing reviews to determine whether you'd want a DevTerm at all regardless of what CPU's actually in it. Instead, I'm going to talk about this device specifically as a RISC-V general purpose computer, either if you'd just like a RISC-V machine to play with or to truly use as an alternative system on the road.

So, about that form factor. Although most people liken the DevTerm to the Tandy Radio Shack TRS-80 Model 100 (the most famous member of the Kyotronic 85 family), actually its design cues come more from the Model 100's close relative, the NEC PC-8201A. (Dig the control diamond: don't tell me that's a coincidence. For some reason CPi chose to make its codes separate from the cursor keys.) But I've used my PC-8201A on the road, for an entire month on Penang Island in Malaysia in 2000 where it was my only computer, and its full-size keyboard made it quite liveable. The DevTerm's a bit ... smaller — 65 percent regular size, to be exact. I have thin fingers (even when I was 45 pounds heavier) and wide hands, and I can sort of touch-type on it, or lift it in two hands and two-thumb-type with my hand width allowing my thumbs to just meet in the center of the keyboard. However, if you lack either of these attributes you may not enjoy the experience, though it has enough ports you could still potentially use it as a desktop system with external input devices instead. (The DevTerm keyboard is also not backlit, but I'm not going to ding it for that at this price point.)

Obligatory unboxing:

The box clearly advertises what's in it and, interestingly, the amount of RAM on the board (at 1GB this seems an odd thing to brag about).

Famously, it comes unassembled. Everything is in nice neat trays and it's actually rather fun to unpack. The plastic standoffs and holdfasts on sprues give it a delightful model-kit feel.

What's not included are a USB-C charger and two 18650 Li-Ion batteries. Neither of those is expensive or tough to find, but you'll need one or the other to actually power it up. There is also no paper for the thermal printer (!) it carries onboard, though any office supply store will have that too.

Case design, schematics and connectors are all GPL and on Github. The RISC-V "core," as CPi calls their CPU modules, sits on anti-static foam in one of the compartments:

CPi modules contain basically everything but storage and peripherals. In particular, the CPU/SoC, GPU (if present) and RAM are provisioned on a 200-pin SO-DIMM and can be swapped out (naturally you'd need to change the operating system at the same time, but that can be as simple as another microSD card). This is a strength of the design because if you don't like the experience with one CPU card (too slow? not compatible?), try another. CPi sells them inexpensively; if you already own an ARM DevTerm you can just buy the RISC-V module for US$29 and download the software. CPi kindly included a RISC-V build of their Ubuntu-based OS with this unit.

In this case, the CPU is an Allwinner D1, a single-core 1GHz RV64IMAFDCVU CPU based on the XuanTie C906 with on-board 2D graphics, DSP, audio/video, USB and SDIO. The DevTerm exposes HDMI, audio and USB external ports directly serviced by the D1. This is the same part in the Sipeed Nezha evaluation board and in a forthcoming SBC from Pine64. Impressively the RTL for the C906 core is open source and Apache-licensed, but unfortunately the rest of the design isn't: in particular the onboard graphics, DSP and peripheral controllers are only available as blobs, and only then if you register as a developer with Allwinner. For CPi's purposes this isn't a killer because their other CPU options are also blobularly blobtastic, but it's a minus in our book. The Hynix chip next to it is the RAM, a single gigabyte as stated. I cannot find any L2 cache at all, just the 32K L1 caches each for I+D.

Another minus is that the C906 and related cores, while they advertise vector instructions, predate the current RISC-V vector instruction standard. The instructions are similar but they are neither binary nor source compatible. On the other hand there's hardly anything supporting the current standard anyway, so this may not be a problem in the long run if the install base becomes large enough.

Getting out the clear orange scaffold and installing the screen, here's a size comparison against a DVD case to give you another idea about how big it is:

As you can see they're roughly the same size. If you'd find typing on a keyboard about half the size of a lengthwise DVD case difficult, the DevTerm is probably not for you. Anyway, let's finish putting it together.
Ta-daa! It took me about half an hour and it was pretty easy with no tools required. The manual goes step by step though they use a lot of part code shorthand that needed some flipping back and forth to check exactly what standoff, etc., I was supposed to be using where. While most of it can be taken apart as easily as it came together, some are glued stick-on components like the little on-board speakers and most notoriously the wireless antenna. I also didn't like the fact the flex cables are a little nervewracking to install and it took a little fiddling to convince myself they were properly seated (plus the video cable in particular has "SCREEN" silkscreened on it, but that end does not connect to the actual screen; it connects to the port marked "SCREEN" on the mainboard). Although it directs you to insert tiny screws to affix the core module to the board, there didn't seem to be any holes drilled on my board and the SO-DIMM socket seemed to hold the core just fine anyway. This might be something specific about the RISC-V module because the assembly manual is generic in scope. Finally, a pro-tip: it's easiest to put in the microSD card during assembly; it felt like I was inserting the card into an empty hole when I did it after, even though it did go in securely.

All that aside I give overall high marks to the fit and finish of the case, though it took me a little time to get the top to mate with the front lip of the bottom. And then there's those Frankensteiny Princess Leia earmuff closures on either side of the screen: they're cute and give it some personality, and they do hold the unit together, and I guess they're better than thumbscrews, but you almost expect them to have some sort of input device functionality and they don't. Missed opportunity, in my opinion. Plus, if you reopen the case the two halves of the closures come apart and have to be snapped back together, which is a little irritating. Once it is together, though, the case feels very sturdy. I liked how it felt in my hands; it didn't feel flimsy or fragile, and it was not excessively heavy even with the batteries in. Total weight according to my kitchen scale is 588g, or about a pound and a quarter.

Booting up! The manual warns it may take up to a minute, which wasn't too far off. The screen is very bright and legible, even considering the 1280x480 resolution is a little odd by modern standards (basically two VGA screens side by side), but it's very glossy and picks up hairs and fingerprints like a magnet. You probably want to have a microfibre cloth around in your bag for this thing.

And booted into CPi's bespoke Ubuntu variant, ClockworkOS. Despite the wiki (it's correct in the on-screen readme), the default username and password are both cpi.
Proof of uname:
ClockworkOS is very lightweight, and thank goodness it is for reasons we'll get into. There's no Wayland crap here; this is Xorg, as G-d Himself intended (especially because — as our frustrations with Wayland on our BMC-only 2D framebuffer Blackbird have proven — Wayland generally does not do well without a GPU even though it's doing better these days than it has). If the window manager looks throwback, it's because it's good old school twm. Fortunately the thumb trackball isn't too bad but there are key combinations for several marquee apps, which I found to be a thoughtful touch.

The status monitor on the right is also very handy, but you'll notice the clock is wrong, and the reason the clock is wrong is because it couldn't contact any time servers. Despite manually creating a network configuration for the house Wi-Fi, the DevTerm wouldn't connect from the front of the house (the Wi-Fi access point is in the server room in the middle) even though every other Wi-Fi capable device I own is able to do so. I was so perplexed by the range I ended up disassembling it again to check if I'd damaged the antenna or if it had come loose when I turned it over to put on the bottom, but the antenna was physically intact and the connector was snapped securely onto the mainboard. The only way it would connect was if it were closer to it. Various Wi-Fi issues have been reported with the DevTerm's relative GameShell, which appears to use the same sort of antenna, though I'm not sure if this is the same specific problem.

There are lots of fun pieces of software pre-installed like DOSBox and Chocolate Doom (and things like GIMP, Inkscape and Xfig if you want to do real work), so I fired up Doom because of course it plays Doom to get an idea of performance. The CPU immediately showed as pegged in the status monitor, which was not encouraging. Neither was gameplay:

I mean, the poor thing's even using a smaller viewport (by default: it started up that way) and the CPU is still straining at 99%. The framerate wasn't slide-show-slow and music and sound effects didn't seem to suffer (use Fn+the volume key to turn up the audio), but you could clearly see it painting each frame.

Web browsing was equally disappointing, but also for different reasons. Firefox on RISC-V is still a work in progress, although there was an Fx94 build at one point; there are patches for Chromium 104, though I wouldn't be caught dead using Chrome or anything derived from it, and neither Firefox nor Chromium are installed in any case. But it does have Qutebrowser, so let's try ... uh ...

Um, okay, how about ELinks?
This worked and might even be more appropriate for the display and CPU anyhow. It was very sprightly. Text for the win. I did go back to Qutebrowser and try QtWebKit despite the warning, and that does start, but ...
Besides the bad rendering, it was also just as slow as playing Doom was. At this point it would seem most appropriate to get an idea of how much oomph this thing actually has. For this we'll use CoreMark, since it's simple, easy to port and verifiable, and it's what many RISC-V vendors cite so this gives you comparison points. ClockworkPi kindly included development tools making it as simple as cloning it from Github and running make.
A couple benchmark results first, using default settings and reporting the highest score obtained: my NetBSD Macintosh IIci (25MHz Motorola 68030, no cache card) gets 8.3 iterations/second, my NetBSD Mac mini (1.5GHz PowerPC G4 7447A) gets 6073.9 iterations/second, my M1 MacBook Air gets 31713.3 iterations/second (single thread) and 171848.9 iterations/second (8 threads), and this dual-8 SMT-4 DD2.3 Raptor Talos II gets 14367.8 iterations/second (single thread) and 430078.6 iterations/second (64 threads). The D1 clocks in at ... 2232.5 iterations/second, just over a third of the performance of the G4 Mac mini, and I can run TenFourFox on that.

Another thing that performs badly on this device: shutting down. It takes literal, honest-to-goodness multiple minutes to power off cleanly. A surprise was the loud "click" it makes when it finally finishes. There doesn't appear to be an obvious way to make it sleep or suspend, though there is a screen saver which turns off the display after a period of inactivity.

I imagine some of these software problems will improve in later iterations, but as it stands this was my out-of-the-box user experience right now (in fairness CPi does say it is a "highly experimental" model, and actively steers new users away from it). I suppose you could try other distros but they may not support (fully) the DevTerm's onboard devices.

Now, something positive: battery life and power consumption seemed really good. The CPU may be weaksauce but it doesn't put a lot of power demand on the hardware either, nor is there a GPU to draw more juice, and the backlight can be easily changed with key combinations or terminal commands. And although there's a tiny little fan on the board I never heard it unless I put my ear right up against it and the device never seemed to get hot even when it was being held at its limits (which was a lot of the time). While the status monitor will show battery percentage (second from the bottom, above the uptime), cat /sys/class/power_supply/axp20x-battery/uevent will display a fuller, different set of statistics that don't always agree. Either way it's thrifty enough I'd estimate you can probably get eight, maybe 10 hours of runtime or more out of a full charge depending on load and battery quality. On the Kill-A-Watt the USB-C charger pulled between 11 and 14 watts depending on CPU load, though even with the CPU pegged in Chocolate Doom it only occasionally drew at the upper end of that range. Incidentally, since the batteries are removeable it's probably more efficient just to shut it down, take them out and stick them in a wall charger at the end of the day.

You might think after all the complaints I've made that I don't like this device. That is absolutely not the case; in fact, I've already become rather fond of it. I'll even go so far as to say that if you want an easy way to try RISC-V and want one you can use like a general purpose computer, and you're not already drunk on the Kool-Aid, then the DevTerm R-01 is your ticket. Clockwork Pi should be commended for offering it and charging less for it on top of that. Offered the choice between a HiFive Unmatched system and the DevTerm R-01, even considering the Unmatched will be somewhat more powerful, I'd still pick the DevTerm. Besides its obvious space and price advantages it's at least got enough grunt to serve as a terminal and do some very basic tasks and do it for hours on easily replaceable batteries, and it comes with sufficient developer tools out of the box that you can test your software on actually available RISC-V silicon today. Plus, with HDMI, USB and Bluetooth, you can just dock it as a desktop system if you don't need it to be mobile. While I certainly had my share of software problems, I suspect they are not at all unique to this particular implementation.

No, my objections here are primarily to the Allwinner D1. For as many claims as RISC-V's proponents make about openness, this chip isn't meaningfully so, and its underwhelming performance doesn't make it worth putting up with. I realize it's aggressively low-end but for crying out loud, it's getting its clock cleaned by a value-spec 2005 Power Mac (the U740 in the Unmatched gets by the G5, but by less than you'd think), and some software still doesn't work on the architecture yet — certainly more so than OpenPOWER. About all it's got going for it is that it can clearly do very well in a portable, power-constrained environment, and it's cheaper than ARM would be in that setting, yet despite such obvious shortcomings it and its progeny are the very chips here and in other upcoming products simply because they exist and survive. Is RISC-V going to be perennially bringing up the performance rear? (Hey, Raptor, want to make a souped-up Arctic Tern in this form factor?) Not for nothing but I don't see anyone selling those ballyhooed Micro Magic parts, and they may well be snake oil. Maybe some future RISC-V system will have sufficient performance, low power usage and full auditability to become a new and self-sustaining libre mobile option, but I'm having a hard time seeing any such CPU on the horizon.

The bottom line: the DevTerm R-01 is fun to play with and makes a stellar introductory RISC-V general purpose computer at a decent price that you can, for some values of "use," use. If you're already a DevTerm owner a measly US$29 for an R-01 module is a slam dunk, and even as a first-time buyer I feel my US$240 wasn't ill-spent. But after this first personal taste of RISC-V, I don't think OpenPOWER has much to worry about right now.