Blue Green Deployments with Azure - The Simple way

4351
Blue Green Deployments with Azure - The Simple way

Every developer goes through the scenario of pushing out a dodgy release to production at least once in their career. Some of us have also experienced a burn so bad it's got our heart pumping as we diligently try to figure out what's wrong whilst having the entire senior management team standing behind breathing down our necks and wanting updates … that’s not a memory you gonna loose anytime soon!

For those of us that have thankfully there is a solution and it's called "Blue Green" (why they call it blue green is a little mystery that a 2 minute googling failed to answer). Blue green is best described in the words of Martin Fowler:

The blue-green deployment approach does this by ensuring you have two production environments, as identical as possible. At any time one of them, let's say blue for the example, is live. As you prepare a new release of your software you do your final stage of testing in the green environment. Once the software is working in the green environment, you switch the router so that all incoming requests go to the green environment - the blue one is now idle. - [1]

Implementing Blue Green can actually be done in many ways, the method described below will be specifically for Azure hosted sites utilising deployment slots, runbook, and TeamCity (although you can use anything you want for this step as it's just going to fire a request for a URL) to trigger the deployments. Anyway enough with the boring talk… on with the tutorial!

Step 1 - Setting up your deployment slots.

 If you’ve never used deployment slots [2] before now's a good time to get started, they're a great way to split out the different environments (commonly testing/staging/production), but also contain them within one web app. Each deployment slot acts as a separate website and has its own configuration.  

How to create a deployment slot:

Azure Portal > App Services > Deployment Slots > Add Slot.

Tip - You can clone a slot from a previously created one which will save time in copying over application settings, shown in the picture below:

For our setup we will need the following slots:

  1. Testing
  2. Staging 
  3. Production Shadow

The production-shadow environment is actually the website that’s going to be swapped with production environment. So its configuration (app settings, connection strings etc) should be identical with production. One key thing to note here is when clicking on "Application Settings" for production shadow each setting including the connection strings have a checkbox called "Slot Settings". This option allows you to persist the setting to the deployment slot so when swapping these settings will remain.

Great that’s the first part already done!

Stage 2 - Creating your runbook to initiate the swap

Azure has a couple ways for manipulating assets on the their portal, one of these methods is by using a runbook which is essentially a PowerShell script runnable (hope this is a word) on azure. The other method is to use Microsoft management library currently hosted on GitHub [4]. However at the time of writing this article the project is in beta and documentation is quite slim. In addition to this you need to handle authentication with Azure manually. Since what we're doing is not massively complicated and won't require much change in the future were going to stick with a PowerShell runbook.

Azure Portal > Automation Accounts > Runbook > Add a runbook

Once that’s created paste in the following code:

workflow Website-Slot-Swap
{
 param
 (
 #Request information contain swap details
 [object]$WebhookData
 )
 
 if ($WebhookData -ne $null) {
 $body = $WebhookData.RequestBody

 Write-Output "Body Print"
 Write-Output $body

 $swapParameters = ConvertFrom-Json -InputObject $body

 Write-Output "All Parameters"
 Write-Output $swapParameters

 $azureWebsiteName = $swapParameters.AzureWebsiteName
 $slot1 = $swapParameters.Slot1
 $slot2 = $swapParameters.Slot2
 
 Write-Output "Swapping slots for website $azureWebsiteName"
 Write-Output "From $slot1 To $slot2"

 $subscriptionName = Get-AutomationVariable -Name "SubscriptionName" 
 $subscriptionID = Get-AutomationVariable -Name "SubscriptionId" 
 $certificateName = Get-AutomationVariable -Name "CertificateName" 
 $certificate = Get-AutomationCertificate -Name $certificateName 

 Set-AzureSubscription -SubscriptionName $subscriptionName -SubscriptionId $subscriptionID -Certificate $certificate 
 Select-AzureSubscription $subscriptionName 

 Switch-AzureWebsiteSlot Name $Using:azureWebsiteName -Slot1 $Using:slot1 -Slot2 $Using:slot2 -Force 
 }
 else {
 Write-Error "Runbook mean to be started only from webhook."
 }

 # Output final status message
 Write-Output "Completed swapping websites"
}

Whilst it may look like a lot of code what it does is quite simple, so let's break it down. As we are going to trigger our runbook through a webhook we have started the script by accepting a parameter with the type of:

[object]$WebhookData

The web hook data will contain the POSTED request body it's in this body that we are going to pass through what website app to swap and which slots to switch over. Before we do the switch we need to authenticate with Azure, as we are using PowerShell authentication is very easy as default parameters are provided for us so it’s a simple case of adding in the following snippet:

$subscriptionName = Get-AutomationVariable -Name "SubscriptionName" 
$subscriptionID = Get-AutomationVariable -Name "SubscriptionId" 
$certificateName = Get-AutomationVariable -Name "CertificateName" 
$certificate = Get-AutomationCertificate -Name $certificateName 

Set-AzureSubscription -SubscriptionName $subscriptionName -SubscriptionId $subscriptionID -Certificate $certificate 
Select-AzureSubscription $subscriptionName 

The code to handle the swap itself is a simple one liner (expecting something more complex  right? As crazy as the syntax of PowerShell is you gotta love those one liners).

Switch-AzureWebsiteSlot Name $Using:azureWebsiteName -Slot1 $Using:slot1 -Slot2 $Using:slot2 -Force 

And that’s it the runbook is created, be sure to test it out before moving on (you can do this within Azure as they have created a handy debug console).

Stage 3 - Adding your web hook

We could leave that runbook as it is and trigger it off manually, but as true dev-ops enthusiasts we will automate to the very end! Creating the webhook is really easy, it's just a case of following the Wizard.

When doing this a question does arise, if I can post to azure and do a website swap what's stopping 'Neo' the hacker from doing the same thing?

The security is currently all within the URL which contains a unique token for this webbook. If the token is lost security will be compromised so it's important not to share this. Also be sure to make a note of the URL as once you have created the web hook there's no chance of seeing it again.

The webhook URL will look something like this:

 https://s2events.azure-automation.net/webhooks?token=<<SuperSecureToken>>

Now that this is created go ahead a do a swap. Note that runbooks are executed like a web job so when a swap is triggered it first gets queued (if your triggering multiple sites the queue trigger will be executed synchronously) and then executed, logs are provided for each execution with input data (what's passed into the WebHook data).

Stage 4 - Be weary of the job 

One thing to consider when doing a swap is that if you have hosted your web jobs with your webapps your production webjobs will get swapped into the shadow environment. However is good practice anyway to separate the two. Why does this happen?

Contrary to what Microsoft tell you, a swap does not actually repoint the DNS, instead it moves the filesystem. As web jobs are stored within the websites AppData folder they end up getting swapped too.

Stage 5 - Creating the build step

The final part is triggering the web job from your deployment software (I'm using Teamcity) but as I'm doing this via PowerShell it should still work in whatever tool you use. The code for doing the post looks like this:

Write-Host "Sending Request To Swap Servers"
$url = "%WebhookUrl%"

$command = ConvertTo-Json @{ 
 AzureWebsiteName = "%AzureWebsiteName%"
 Slot1 = "%SlotFrom%"
 Slot2 = "%SlotTo%"
}

$bytes = [System.Text.Encoding]::ASCII.GetBytes($command)
$web = [System.Net.WebRequest]::Create($url)
$web.Method = "POST"
$web.ContentLength = $bytes.Length
$web.ContentType = "application/json"
$stream = $web.GetRequestStream()
$stream.Write($bytes,0,$bytes.Length)
$stream.close()

$response = [System.Net.HttpWebResponse]$web.GetResponse()
if($response.StatusCode -eq [System.Net.HttpStatusCode]::OK) {
 Write-Host $response
}

Stage 6 - Brush your shoulders off

It's important to stand back and admire your slick new process you put in place, especially considering that it's got the power to save you’re a** when a dodgy bit of code slips through the safety net (its only ever a matter of time). 

Links and References

  1. https://www.martinfowler.com/bliki/BlueGreenDeployment.html
  2. https://docs.microsoft.com/en-us/azure/app-service-web/web-sites-staged-publishing
  3. https://docs.microsoft.com/en-us/azure/automation/automation-intro
  4. https://github.com/Azure/azure-sdk-for-net/tree/Fluent