What to do when the BMC won't talk to you

Normally I'm typing TenFourFox posts from the Talos II, but today I'm typing a Talospace post from the old Quad G5. When my dual-8 second T2 arrived, it turned out the onboard NICs were duds (I'm not blaming Raptor for this, mind you; crib death happens with any system). When I started moving components off it to RMA the motherboard, however, one of the RAM sticks got jammed in my old T2 and when I tried to free it, the latch not only snapped off but with such force that it struck the NIC ROM chip and knocked one of the surface mount resistors connected to it off. I will simply say that the cat fled behind the washing machine and didn't come out for hours after I realized what had happened.

That untimely accident thus left me with two systems with dead network ports, and the BMC, the heartbeat of your OpenPOWER machine, the breath of life, the starter, the prime mover, only talks on one of them. Sure, I could plug in an (ugh) USB Ethernet dongle (protip: the old TrendNET USB NIC I dug out of storage doesn't work in Petitboot, so I had to unplug and replug it when Fedora booted), or waste a PCIe slot on another NIC or whathaveyou, but none of those things restores connectivity to the BMC that for security reasons is limited to that top onboard port. The BMC is now an island unto itself, never to be updated or talked to again.

Or ... is it? The BMC fortunately has a serial port. With an easily available 10-pin to DE-9 cable which you can get on eBay for a few bucks, you can talk to your BMC again.

There's no nice web interface (I'll have a later post about how to use TenFourFox and its built-in AppleScript-to-JavaScript bridge to support its later features if you're a Power Mac refugee like me) and everything is strictly by shell, but if you were already used to talking to your BMC over SSH then this won't seem at all unfamiliar. Here it is connected to a USB serial dongle back to the G5.

In ZTerm on this G5 I set the serial connection to 115200bps, 8-N-1. Assuming your system is running the 2.00 system package or later, hit RETURN/ENTER a couple times and you should get a "Phosphor" login prompt; log in as root and your BMC password. You'll get to the root prompt just as you would over SSH.

That suffices for logging in and changing your password if necessary, but what about updating the firmware? Fortunately the clever people who maintain OpenBMC, the standard firmware in Raptor and most other OpenPOWER machines, added rz and sz to the build. That means you're just a ZMODEM upload away from updating your BMC and PNOR! Just go into the correct directory (/tmp for PNOR, /run/initramfs for the BMC firmware), run rz from the shell, and then start a ZMODEM upload from the terminal program on your assisting computer (to update the BMC, you'll upload both image-kernel and image-rofs, and to update the PNOR, whatever file as named). Some terminal programs, or the sz utility if you use it on the client side, will allow you to specify an absolute path. Either way, once it's there, you can then finish the update the usual way.

The Blackbird has such a port as well and uses the same cable as the T2 and T2 Lite. You can even leave the cable connected and this will allow you to access the BMC at any time. Note that rebooting the BMC while the main processors are still running may cause your machine to act a little funny until they're back in sync.

As a postscript, those SMT resistors are damn small, and I'm pretty sure it's not right on the pads anyway assuming I didn't damage something else in the process. But the RMA board did arrive and does work, so I'm rebuilding the new system and we'll talk about whether the jump from a dual-4 to a dual-8 T2 is worth it when I do. The old system still works otherwise and at least with that serial cable and a quad-port PCIe NIC, it may be one slot less but I can still put it to good use.