Hey Folks,
I went to a clients site to get their server online after an outage. (power failure for a few hours, so I went in to make sure the server came up smooth as this is a pretty new client). A few weeks earlier I was at the site for a problem with the domain controller VM freezing at a black screen. All I had to do was reboot the VM and it came up fine. After looking at the logs and doing some searching I found out that VM freeze was specific to the type of motherboard they had (intel) and the exact CPU they had. There was a known firmware bug that caused VM's to occasionally freeze with this combination of hardware and specific to 2008 R2 VM's (perfect storm!) Intel released a firmware update which included a microcode update to the CPU to resolve this.
So, since I was already down there for an outage, and we were scheduled to come down the following day to install the firmware update, I decided to do the BIOS update while I was there. I shut down the VM's and the host server and rebooted the host off the Intel Deployment assistant DVD. The dvd connected to the internet, found the BIOS update and downloaded it. after installing it said it needed to reboot, this is where things went south. After reboot I was met with a black screen on the host server (no splash screen or anything else indicating its posted).
I tried to move the jumper to the "recover bios" option, this got some text on the screen showing BIOS information, and a blinking cursor, but nothing else. I tried downloading the recovery BIOS form intel, but couldn't get it to load (I just saw this cursor, it never seemed to attempt to read from the USB stick with the recovery BIOS).
I also tried clearing the CMOS and even tried pulling the battery for 10 minutes to let it reset itself. All were met with the same black screen. After phone calls to my vender, it was determined that the board is bricked and a new one is coming over night.
Anyone have something similar happen before? I've been in this game for about 12 years, and have heard plenty of these horror stories but never had it happen to me personally.
As a side question, how do you guys handle firmware updates to servers? Typically I never apply a firmware update unless its to fix a specific problem, which happened to be the case here. I figured the intel deployment assistant was the easiest way to apply this (and it was pretty easy... except for the bricking the server part). Any suggestions on how to avoid something like this in the future?
I went to a clients site to get their server online after an outage. (power failure for a few hours, so I went in to make sure the server came up smooth as this is a pretty new client). A few weeks earlier I was at the site for a problem with the domain controller VM freezing at a black screen. All I had to do was reboot the VM and it came up fine. After looking at the logs and doing some searching I found out that VM freeze was specific to the type of motherboard they had (intel) and the exact CPU they had. There was a known firmware bug that caused VM's to occasionally freeze with this combination of hardware and specific to 2008 R2 VM's (perfect storm!) Intel released a firmware update which included a microcode update to the CPU to resolve this.
So, since I was already down there for an outage, and we were scheduled to come down the following day to install the firmware update, I decided to do the BIOS update while I was there. I shut down the VM's and the host server and rebooted the host off the Intel Deployment assistant DVD. The dvd connected to the internet, found the BIOS update and downloaded it. after installing it said it needed to reboot, this is where things went south. After reboot I was met with a black screen on the host server (no splash screen or anything else indicating its posted).
I tried to move the jumper to the "recover bios" option, this got some text on the screen showing BIOS information, and a blinking cursor, but nothing else. I tried downloading the recovery BIOS form intel, but couldn't get it to load (I just saw this cursor, it never seemed to attempt to read from the USB stick with the recovery BIOS).
I also tried clearing the CMOS and even tried pulling the battery for 10 minutes to let it reset itself. All were met with the same black screen. After phone calls to my vender, it was determined that the board is bricked and a new one is coming over night.
Anyone have something similar happen before? I've been in this game for about 12 years, and have heard plenty of these horror stories but never had it happen to me personally.
As a side question, how do you guys handle firmware updates to servers? Typically I never apply a firmware update unless its to fix a specific problem, which happened to be the case here. I figured the intel deployment assistant was the easiest way to apply this (and it was pretty easy... except for the bricking the server part). Any suggestions on how to avoid something like this in the future?
