When Petitboot barfs, everything's vomit

Colourful, no? But it's true. I've not been able to write up my Fedora 37 experience, nor upgrade Firefox (nor do further work on the JIT) because the Petitboot boot menu couldn't stop touching the main NVMe drive and making its older (Linux 5.5) XFS kernel module hang. If Petitboot can't start, your expensive POWER9 system is a brick.

In its most literal sense this article is largely a precautionary tale, because unless you're a long-term Fedora user like me with a continuously updated older installation, it's very unlikely you have an XFS volume in your OpenPOWER box. But if the antique kernel in Petitboot ever starts barfing on your own filesystems or a device you install, you'll be in this state too, so here's how I got the Talos II working again.

It's pretty much been a constant that you need a second system to deal with glitches. For me, this is usually my trusty Quad G5 Power Mac sitting next to the T2 which is connected to its serial port (or to the BMC's), and this works when it's a problem you can resolve from the BMC side, which is many of them. It would be nice to power up a Talos or Blackbird and have the console automatically start up talking to the BMC instead of needing another system to do so but this is what we have, at least until Kestrel develops that capability.

Unfortunately, this wasn't one of those problems:

  [Disk: nvme1n1p2 / 19a5d4e3-19f7-423f-a75b-5b15c8ee0bff]
    Fedora (0-rescue-ee275f6a7d994c9981e4e1436b83172d) 30 (Workstation Edition)
    Fedora Linux (5.18.13-200.fc36.ppc64le) 36 (Workstation Edition)
    Fedora Linux (5.18.18-200.fc36.ppc64le) 36 (Workstation Edition)
(*) Fedora Linux (6.0.12-200.fc36.ppc64le) 36 (Workstation Edition)
  System configuration
  System status log
  Language
  Rescan devices
  Retrieve config from URL
 *Exit to shell


 [fedora-root] Processing new Disk device[    8.041704] XFS: Assertion failed: !
(fields & XFS_ILOG_DFORK) ||
 (len == in_f->ilf_dsize), file: fs/xfs/xfs_log_recover.c, line: 3103
cpu 0x26: Vector: 700 (Program Check) at [c0002007e33171c0]
    pc: c008000008dc46bc: assfail+0x54/0x60 [xfs]
    lr: c008000008dc4694: assfail+0x2c/0x60 [xfs]
    sp: c0002007e3317450
   msr: 900000000282b033
  current = 0xc0002007e32c3180
  paca    = 0xc0002007ff7f5900   irqmask: 0x03   irq_happened: 0x01
    pid   = 649, comm = pb-discover
kernel BUG at fs/xfs/xfs_message.c:110!

After the assertion appeared, Petitboot locked up (at least on the regular console) and the system wouldn't start from any device because Petitboot could not be coerced into ignoring it. I tried holding down the x key from the serial console to force it into the shell, and that worked — but it still tried to mount the volume anyway and died. This did bring up a live kernel debugging session as you can see in the screenshot, but since I wasn't sure what the XFS module would do at this point and didn't want to risk the filesystem, I just powered it down.

Something about the state of the root XFS volume after the Fedora 37 update was making it go wrong, and I haven't been the first to observe this, either. Recovering cleanly would at minimum require a system that can mount and examine the XFS volume, and the G5, which runs Mac OS X Tiger, isn't that system. (Maybe the SGI Fuel next to it with IRIX 6.5.30 is — though that's something to explore some other time when it isn't my primary computer's boot volume at stake.)

Fortunately I've also got a Blackbird that did complete its F37 upgrade successfully. So it's time to do a little shopping.

I picked up two off-the-shelf NVMe-to-USB enclosures, one the Worst Best Buy Insignia store brand NS-PCNVMEHDE for about US$20, and a Sabrent EC-SNVE for about US$30 which also supports SATA. I was pretty sure the Sabrent would work due to their usual diligence about Linux, but I bought the Insignia anyway as a backstop in case the Sabrent was defective, and also because it came with a USB-C to USB-A converter since the Blackbird doesn't currently have any USB-C connectors.

Both devices are USB 3.2 Gen 2 and came up as "SuperSpeed USB" connected to the Blackbird's rear USB ports. The Sabrent is a much nicer unit with high-quality metal construction that folds open and has an integrated heat spreader in the top. The "tool free" part is there's a small clip that rotates to hold the M.2 stick in (with a stopper in the package for smaller-sized sticks). But even though the Insignia was kludgier (pulls out instead of folds open, requires you to stick on a heat spreader, really clumsy turn clip), it supports USB Attached SCSI Protocol; dmesg indicated the Sabrent didn't respond to a UAS probe. If I could have combined the chipset in the Insignia with the case of the Sabrent, we'd have the perfect enclosure.

Both devices also worked in Petitboot — by which I mean having the tainted NVMe SSD plugged in while Petitboot came up would also crash the Blackbird.

Bringing up Fedora first and then connecting the enclosure after, we next get the T2's root volume up so it can be checked. Because both the Blackbird's boot drive and the T2's boot drive have the volume group name fedora, we'll need to rename the T2's. We list the volume groups with vgdisplay; the T2's starts with lO, so the commands are:

vgrename `vgdisplay | grep lO | awk '{print $3}'` tfed
lvchange -ay /dev/tfed/root

But xfs_repair /dev/tfed/root wouldn't try to fix it: it said there was a log entry that had to be replayed first. This can be done simply by mounting it, so

mount /dev/tfed/root /mnt
umount /mnt
xfs_repair /dev/tfed/root

This showed no errors, so I inactivated the root LV again with lvchange -an /dev/tfed/root, disconnected the NVMe stick, put it back in its PCIe carrier and reinstalled it in the T2. Petitboot didn't crash, but Fedora requires the logical volume be named fedora, so we enter the Petitboot shell first and finish up with

vgrename `vgdisplay | grep lO | awk '{print $3}'` fedora

and then boot.

Whose bug was this? Well, arguably, Fedora might not have properly unmounted the drive after the update, but the error appears to be minor in that simply mounting the drive (with a later kernel, admittedly) fixed up the issue. It's more important that Petitboot have a stable, well-tested codebase, so the decision to use an older kernel (though 5.5 is a little excessive) is not an unreasonable one, and this older kernel appears not to be able to do that kind of recovery.

But if Petitboot can't do it, it shouldn't just brick the system. There should be a way for a user to hold down a key and bypass the menu without mounting anything, and try to recover in the shell at that point, which you can do from the console. Similarly, if it barfs on a filesystem or an installed device, it should simply say so and ignore it, not panic. These computers are just too expensive to have vomit everywhere when something goes wrong — and you shouldn't have to have a whole second system around to clean up the mess.