Messing with the new 2.0 BMC

Tonight's attempt to upgrade the Blackbird to the new 2.0 BMC firmware did not go smoothly, though some of this was self-inflicted, and I'm also still flattened by whatever hellish virus has gripped me for the last month (it's not coronavirus, or at least not that coronavirus) which causes me little tolerance for glitches. TL;DR: the firmware basically works, but when I used it to reconfigure its IP address the BMC now can't see anything and nothing can see it, which has left me in somewhat of a foul mood [but see postscript]. I'll get it working when I'm feeling better and you should probably still update, but beware of these pitfalls.

Updating to 2.0 from any pre-2.0 version requires a complete flash of the BMC. Raptor warns against this generally because all your U-Boot/firmware settings will be reset, but in this case it's unavoidable. That brought us to the first problem: when sshed into the BMC, at the root prompt fw_printenv is supposed to show you the IPMI MAC address so you can reprogram it. On this Blackbird, however, it showed absolutely bupkis except for the serial port settings. After a brief moment of panic I realized I had a picture of the mainboard from the Blackbird semi-review and could enter it from that. Otherwise you'll have to drag the machine out, open it up and jot down the address printed on the board. Oddly, this does not reset anything else, including the BMC password or actual network settings. More about that in a moment. It did, however, change the ssh key.

Now updated, since my other main systems are Power Macs (and what better computer to be a "service processor" to your Blackbird than another PowerPC?), I decided to do further configuration through the BMC's new web interface in TenFourFox, which is essentially Firefox 45 with a lot of patches. These were done on my iBook G4 service laptop running the latest beta hot off the G5 in the backroom.

The first thing to keep in mind is that the certificate is self-signed. No biggie, just expect it.

The webapp appears to be written in Angular, and it's using JavaScript too recent for TenFourFox (which admittedly doesn't get along well with current React or AngularJS frameworks). Some stuff does work -- the IPMI sensor data loads -- but does not automatically update, and the server status never appeared. It might have been nicer to have a better fallback, especially for the NoScript people, so that data can be displayed even if it won't update until one reloads the page.

It didn't appear here either, even when directly queried.

Fortunately my main use case was to upload firmware through the web interface, so I decided to immediately update the PNOR (both out of necessity and as a useful test), and that worked. Just unpack the archive and upload the subarchive in the web_ipmi folder (the server will automatically unpack the .tar.gz and make the firmware available). TenFourFox threw a weird error at the end but the firmware uploaded, was verified, and could be activated.

IPL, showing the fans coming up. You can boot through the interface, but I just pushed the power button since I was sitting next to it.

The serial port output did not work on TenFourFox either, so I did it from Firefox on the MacBook Air, which I found technically disgusting but worked rather well. Fedora will happily run on the serial port. I was able to log in and look around from the BMC itself. Yes, using TenFourFox was a self-inflicted wound, but I thought it would have worked better than it did.

At this point I decided that I'd had enough mucking around with the Blackbird over WiFi and decided to give it a new static IP through the web interface and run it to the iBook over Ethernet. I did this from the Air, just in case the iBook screwed it up. The machine obligingly accepted the settings and then stopped responding on any address, even after a power cycle. Tomorrow I'll try to find a serial connector to talk to the board directly and try to start over from scratch. I would have your network settings finalized first before this update, as you probably should anyway.

I haven't tried doing this update on the Talos II. I might not anyway since my tax refund should be arriving and I'll be upgrading to a dual-8 system soon. I can't imagine there's much difference in firmware experience between the two systems, though.

The moral of the story is don't update firmware when you're ill.

POSTSCRIPT: ipmitool saves the day! After an obscure mention in an IBM technical manual I was reading for another purpose, it dawned on me that Petitboot (or, for that matter, Fedora) can set the BMC's address.

I started up the Blackbird and went into the Petitboot shell. A quick ipmitool lan print 1 showed what the problem was: the new web BMC interface claimed it had removed the old ZeroConf IP address, but had not, and that became the IP. Since the netmask was now all munged, nothing could see it and it couldn't see anything. So I forced the issue:

ipmitool lan set 1 ipsrc static
ipmitool lan set 1 ipaddr [your IPv4 address]
ipmitool lan set 1 netmask [your IPv4 netmask]
ipmitool lan set 1 defgw ipaddr 0.0.0.0

A quick powercycle confirmed it stuck, and the web BMC answers correctly on the expected address. I still consider this a bug in web BMC but fortunately it's recoverable without digging out the serial cable.