Azure Stack – It is time to party like it is 2019!

By Azure Stack, Azure Stack Update,

Ok, so instead of going into the details of Azure Stack’s latest release, which is  1.1908.0.20 . I want to focus on how I got my your stamp ready to deploy  1.1908.0.20. However, I do want to touch on a few things about 1.1908.0.20 and why this is a very big release with very limited changes.

In fact, there are really only 2 new changes to this update. However, the two changes we are talking about are huge!!!! So huge that this update requires a lot of things to be in place before you can even move forward with the update itself.

The two new changes and pretty much the only changes outside of security updates and hotfixes are:

  • The underlying operating system on which Azure Stack runs has been updated to Windows Server 2019. This enables core fundamental enhancements, as well as the ability to bring additional capabilities to Azure Stack in the near future.
  • All components of Azure Stack infrastructure now operate in FIPS 140-2 mode.

For those that are not aware, before Azure Stack’s underlying operating system ran on WIndows Server 2016, which means it was limited to features Windows Server 2016 would offer such as SDNv2 and other features that where great for the time. In order for the product team to bring to Azure Stack some of the services we all have been asking for, this update will resolve some of the road blocks. Now, for a tease, there are some very awesome features and changes that will be coming to Azure Stack in the future. Now that everything is running on Windows Server 2019. However, 99.9% of them we can’t talk about because we promised not to. Plus there are things like NDA’s we don’t want to break. 🙂

For a quick view of everything included in this release you can check the release notes for 1.1908.0.20.

First of all, this is classified as a Full update and not an Express update. Which means you need to make sure your maintenance windows for your tenants or internal customers are set for a very long time frame. I am talking about a few days if you include some of the OEM prerequisites work needed.

So this blog will focus on my experience upgrading my Azure Stack stamp to 1908. Fasten your seatbelts and get ready for a ride of your life. Well, actually, it is the PNU process for Azure Stack, which is fairly boring in itself. So, grab a good book, or a good movie, and be prepared to sit and watch update logs for a few days.

Let the fun begin!

So let me begin with how my current stamp’s update level was before I started the process. I did have my Azure Stack version already at the latest at that time. I was currently at 1.1907.12.44.

My current OEM version was a little behind. I was still running Dell EMC’s OEM Extension version 2.0.1901.1. Yes, I know, almost 8 months ago was my last OEM Extension update. However, unless it is mandatory at this point, we will keep the OEM extension updates at a minimum. (unless I get yelled at from Dell EMC after they read this.)

Prerequisites

So the two prerequisites to upgrade to 1.1908.0.20 are:

So I was batting .500 right? I met the requirements for being at the latest Azure Stack version but I was way behind for my OEM version. Not so bad right? Just download the latest OEM Extension package, install and bingo I should be ready for 1908, right? Not really.

I am not sure if it is a requirement now, but we were told to make sure before installing the latest Dell EMC OEM extension that would support 1908, we needed to be at least up to 20.0.1906.1. So I did that, which again is fairly easy to do, it is just like installing the Microsoft PNU manually. It just takes time. In fact, 12 hours and 32 minutes on a 4 node Azure Stack stamp. However, like most experiences with the PNU process, this was an uneventful update which means it ran and it worked.

So now I am actually ready to install the final prerequisite in order to finally get 1908 installed. Now, before I even started, I downloaded Dell EMC’s “Upgrading to Microsoft Windows Server 2019 for Dell EMC Cloud for Microsoft Azure Stack” document which did help with the process along the way. The version I had access to was still in the works. However, the document that is out now is complete and you shouldn’t have an issue following it. I will call out areas I had issues but hopefully you won’t have these issues with the updated document.

Before you start, make sure you are familiar with how to install the OEM Extension packages. You can learn more from the following Microsoft Doc’s reference. Apply Azure Stack original equipment manufacturer (OEM) updates. This link also has links to each OEM’s location where one can grab the most current updates and documentation as well.

The Prerequisites before the Prerequisites

This next step I think is different depending on your OEM. I could be wrong but I don’t think my friends with HPE stamps had to do this and I don’t think my friends with Cisco stamps had to do this. I just know that I had to do this and I have a Dell EMC stamp.

There are two things that need to be done before you can even install the newest Dell EMC OEM Extension package.

  • Run the latest RegisterJEAConfiguration script on OME-VM
  • Add Hardware Manager IP address and credentials in Azure Stack

From the OME-VM you will need to run the RegisterJEAConfiguration script. This script, including the OEM Extension update can all be downloaded from the Dell EMC support site for your Azure Stack. This site does require you to have an account. Download everything and extract it into the folders as directed by the provided documentation. (Note to self…. later on in the blog I will talk about an issue related to this process.)

Run the following command within a PowerShell Administrator session.

Now, this is where I got confused the first time I ran this script. The first prompt is asking for the administrator credentials for this system. I didn’t read that correctly and I used my cloudadmin credentials here. This caused issues when I was finally starting to deploy the OEM Extension update. This is asking for the OME-VM administration account.

This is another area that I ran into issues. This is the second prompt for credentials. After a few attempts I finally just entered a generic user name called jeauser with a generic password. Which worked finally later on down the process. Just make sure you write down the password and user name. You will need it here soon. Also, to be honest, I don’t think you need this account moving forward. To be safe, I stored it in my password vault anyway.

Once you enter the credentials the script will run and reboot the OME-VM a few times. This process will take about 5 minutes or so.

Once that script is done you will need to will need to create a New PSSession and run the following command. They say to do this from the HLH, however I am not sure if that is necessary?

At this point I was prompted for more credentials. Again, with trial and error I found out the hard way that this is asking for the previously created credentials. I entered the jeauser and the password I set for it and continued on.

This process will take some time. I can’t remember how much time but if I had to run it again I would go get some coffee and a donut or something and maybe walk the dog? Kidding, it doesn’t take that long but I think it does run for a few minutes. At the end you should see something like this.

Finally, Install the final prerequisite

Now, it is time. I am finally able to install the Dell EMC OEM Package Extension version 2.0.1908.2 which is a requirement in order to start installing 1908.

So I started it…………. and………..

As you can see I had many failed attempts. One of them was because I used the incorrect user name and password. A few more where related to something very simple. When I downloaded the zip files from Dell EMC and extracted them I never unblocked the original file. To be honest, this has never been an issue with previous updates so I didn’t even think of this. However, with the files blocked, there are steps within the \Register-JEAConfiguration.ps1 script that never completed.

Thanks to some very smart Dell EMC engineers a quick fix was identified. That fix, was to unblock the file and the extract everything. Then rerun the script, etc.

As you can see I finally had a successful installation of Dell EMC OEM Extension package version 2.1.1908.2. Again, the update process took about 9 hours.

So now all the prerequisites are installed and we are now ready to start the 1.1908.0.20 installation process.

Installing Azure Stack Update 1.1908.0.20

So now I am ready to install update 1.1908.0.20. First thing first, I ran Test-AzureStack -Group UpdateReadiness first.

All is good. This is an important step which I have seen a few people, including myself skip. It can come and bite you in the behind when you try to start an update if something isn’t healthy.

So I uploaded the most recent files to my stamp and within a few seconds I had the notification that an update was ready to install.

Now a few days later and a few conversations with very smart guys at Microsoft we finally had a successful installation of the 1.1908.0.20 update.

Now to be transparent, my 1908 update didn’t go very smoothly. In fact, this was the first update where I was actually worried about tenant workloads. We really don’t have anything production worthy running right now but still, I was worried about the workloads I did have running. At the end of the process all my workloads never went down. I did run into issues with both the admin and tenant portals, they both stopped working completely. These issues were all related to 1 of 2 issues that I ran into during the update process.

Issues Along The Way

As you can see I had at least one issue during the 1908 update below. However, with the assistance of the team at Microsoft we had this issue identified and resolved very fast.

The first issue was during the Dell EMC OEM Extension upgrade. During the process, when the node was being redeployed after the firmware installation. We ran into an error that Node 03 couldn’t be drained because of some kind of connection issue.

What really was happening was Node 02 was still offline, in a stopped state, and still in maintenance mode. Once I jumped on the iDrac for Node 02 it was clear the Node was completely powered off. Something during the eviction process of the node powered it off. So basically the entire process timed out waiting for it. With a quick Power On using the iDrac, Node 02 went back into the process of updating firmware. Then the job continued on like it should and about 8 hours later we had a successful deployment of the lates OEM Extension package.

The last major issue we had was during the Microsoft PNU process of installing 1.1908.0.20. I have to give kudos to Steve Linehan from the Microsoft team for working with me on a Sunday, and for identifying the issue fairly fast and a fix even faster.

This issue was the reason pretty much everything wasn’t working. My portals stopped working because of this, etc.

At a high level, during the redeployment of the 2 internal SQL servers, the SQL cluster basically crashed. SQL01 had been evicted to be rebuilt however, SQL02 which became the primary had some issues communicating to the DNS servers. So, long story short, we rebooted SQL02, the cluster witness came back online because it could now communicate with the file share. The SQL instance came back online because now the witness was working and SQL02 was able to resolve names. In turn, at that moment my portals came back online because the sql cluster name was now working again and those instances could now communicate to the SQL servers using the now working SQL cluster name.

We resumed the update and had a successful update. After a good 4 or 5 days of getting all the prerequisites installed and working we finally had 1.1908.0.20 installed.

Final Thoughts

It took some time to finally see this:

However, with the great work from both Dell EMC and Microsoft everything came together at the end and now our Azure Stack stamp is running the latest and greatest version of Azure Stack. I am looking forward to future features that can now come down to stack with the underlying OS now being Window Server 2019. I know not all updates will be this intense and so far my experiences overall has been very awesome with Azure Stack updates. Keep up the good work Microsoft and the entire Azure Stack team!

Spread the word. Share this post!