Saturday, 14 January 2017

Upgrade to ESXi 6 failed - Upgrade option is missing in the consecutive attempts

Recently I had an issue while upgrading the customer's environment from ESXi 5.5 to 6.

This was very sensitive vSAN cluster with numerous issues so I had to manually upgrade hosts.
One of the hosts failed during the upgrade process with an error "[Errno 28] No space left on device"

After some troubleshooting I found that the /locker/packages folder contained 5.5.0 and 6.0.0 packages folder so I moved both these folders to a shared datastore to cleanup up some space.

However, when I tried to run the upgrade for the second time the installer didn't provide Upgrade option. If you open the details of the disk where ESXi is installed, in my case SD card, you will see that the Installer cannot find ESXi there.


However, I could still boot ESXi host just fine. 

Well, the thing is that ESXi contains two boot partisions with two symbolic links to boot partitions /bootbank and /altbootbank.
When ESXi is updated/upgraded the new files are actually writtent to the /altbootbank partition and then the symlinks are updated so that /altbootbank partition becomes /bootbank partition and vice versa.

That allows to rollback the ESXi update/upgrade if something goes wrong with the /bootbank. 

In my case the /altbootbank wasn't fully updated due to the failed upgrade process and it didn't containg the state.tgz file which is actually a collection of configuration files. Some othere files were missing too and the sizes of two partitions differ significantly. 

So, it looks like when the /altbootbank is corrupted and doesn't contain all files the Installer refuses to recognize installed ESXi.

Therefore, I deleted all files from the /altbootbank partition and copied the content of /bootbank over and on the next attempt I was suggested to upgrade the ESXi host. 

3 comments:

  1. I have this same issue. Could you give me a step by step on how you copied the bootbank files to the altbootbank directory please

    ReplyDelete
    Replies
    1. For anyone wondering how to accomplish the steps suggested in the article:

      #SSH to the ESXi host
      #List Files
      ls
      #Verify you see an altbootbank and a bootbank directory
      #Move into altbootbank
      cd altbootbank/
      #Remove everything in here.
      #confirm you are in /altbootbank/
      pwd
      #WARNING YOU ARE ABOUT TO DELETE FILES BE CAUTIOUS AND PERFORM BACKUPS IF NEEDED
      #BE SURE YOU ARE IN THE RIGHT DIRECTORY BEFORE RUNNING NEXT COMMAND
      rm *
      # Perform listing to confirm the dir is now empty
      ls
      #Move back to root and jump into bootbank
      cd ~
      cd bootbank
      #Perform listing to check what files exist
      ls
      # Copy files to altbootbank
      cp * /altbootbank/
      #Perform a listing in altbootbank and compare against the ls you ran in bootbank above
      ls /altbootbank/
      #Perform your next upgrade attempt

      Delete
  2. This comment has been removed by the author.

    ReplyDelete