Swap workflow didn't actually swap the Web App Slots (OSOE-1113)

[BenedekFarkas]

This is likely a problem in Azure, but adding details here for now in case we find a way to detect such issues in the workflow itself, e.g. grabbing the version number of the app deployed and comparing it to the one previously deployed to Staging, which is what's going out to Production.

This Swap workflow instance shows the DotNest Tenants Core was swapped out from Staging to Production, even Azure reports that it happened:

In reality, the app code in those environments remained the same, so nothing happened. Unfortunately this messes up the automatically created/moved tags, so those changes had to be reverted manually.

Restarting the app didn't help, but a repeated a execution of the workflow worked as expected.

Jira issue

[Piedone]

In my understanding, the issue is about having a way to verify if the swap happened (and fail the workflow if it didn't), rather than the swap not happening due to an error in our workflow (because as far as possible to tell from the Azure API, it happened and our workflow did what it's supposed to), right?

That BTW I don't know how we can do that nicely. We could add some kind of API to the app that retrieves at least its version (as our module does when showing it on the admin). Or we'll need to access its binaries to read that out. That is, if Azure's API also lies about the swap happening and we can't trust that.

Although at this point, this is rather something that we need to report instead of trying to work around.

[Piedone]

@BenedekFarkas?

[BenedekFarkas]

I don't have much faith in the the Azure support, so let's see if this happens again and then they might have better chance of finding something useful (it's been a while since this happened).

What I'm seeing in the EventLog is Failed to gracefully shutdown application type of errors 1-2 minutes apart for the Production slot, but then a few minute later the main application process is shut down. Those failures might get it in the way of the swap completing, but hard to tell.

Otherwise some way of checking the running app version will be necessary to verify that the swap went through.