ASM to ARM migration and fun with Key Vault

Update: This is now possible, yet billing issues might occur. See this post: https://cloudpuzzles.net/2016/12/update-asm-arm-migration-experience/

TL;DR: ASM to ARM to new subscription, is currently not possible without downtime.

The rant:
As you might know, a lot of stuff is going on in Azure. Not only technical stuff, but also from a payment/partner perspective. A short history:

Azure started out as a Pay-As-You-Go (PAYG) service, and later on introduced Enterprise Agreements (EA) and Open Licensing. Those deals bypassed the partner channels (somewhat at least), which obviously wasn’t satisfying for those partners. Enter Cloud Solution Providers (CSP) where partners can resell Azure, O365 etc., directly to customers. Over the past year I’ve been working with companies (license houses, consultants etc) to teach them how to get started on CSP and onboard customers.

So the scenario is: Customer is running Azure, using Azure Service Management (ASM) which is also known as “Classic”. Now they want to buy Azure through a reseller: CSP. Buying Azure through CSP, you can’t use ASM. It’s simply not available (more info: https://cloudpuzzles.net/2016/02/buying-azure-through-cloud-solution-providers-csp/), so we have to migrate to Azure Resource Manager (ARM) first, and then move resources to the new subscription afterwards.

From an infrastructure point of view, this has been a huge PITA. We’ve used tools like Double Take Move from Vision Solutions to avoid too much downtime. Those license cost money though, so other customers was migrated offline using PowerShell. More info on that part here: https://cloudpuzzles.net/2016/07/notes-field-migrating-azure-classic-azure-resource-manager/

I’ve used the Platform Supported Migration Tool from Microsoft the last couple of months, to migrate from ASM to ARM. Mainly to prepare customers to for CSP migrations later on. The big deal here, is that there is no downtime involved. Everything is migrated on the fly. That was the easy part, ASM to ARM… Next step is moving to CSP subscriptions for those customers. Ready for a headache?

There is a built in tool in Azure, to move resources between resource groups (more info here: https://cloudpuzzles.net/2016/07/move-azure-vms-vnets-storage-accounts-resource-groups/). Again, no downtime is involved, it’s perfect! Using PowerShell we could also move to new subscriptions (this has since been added to the portal too), but there is a blocker: Key Vault
“Why do I all of sudden have a Key Vault?” You may ask. The thing is, Cloud Services (an ASM concept) have certificates assigned to enable WinRM etc. on public IPs. Those certificates (1 per Cloud Service) are moved into Key Vaults when migrating to ARM. From there, the VMs depends on the Key Vault – see the highlighted part here, which is the osProfile part of my VM configuration:

vm_config_key_vaultThis means, if you want to move to a new subscription, you have to move the Key Vault with the VMs.. No problem, right? One should think so, since Key Vaults are supported to move between subscriptions: https://azure.microsoft.com/en-us/documentation/articles/resource-group-move-resources/#services-that-enable-move

Well, that’s only half the truth. You can’t move the Key Vault when it’s connected to the VMs. Yes, that is true. Okay, I’ll just delete the certificate then, right? No, you can’t remove the connection between VMs and Key Vaults at the moment. It doesn’t matter if you use the certificate – or WinRM – or not.

If you try to delete the Key Vault, you will get this error message, when you try to move the resources:

Move-AzureRmResource : {"error":{"code":"ResourceMoveProviderValidationFailed","message":
"Resource move validation failed. Please see details. Diagnostic information: timestamp '20160923T124509Z',
subscription id '7bce381f-79b3-4521-b36e-138932735300', tracking id '631f93e8-9065-490d-91cf-26925d6b1bf5',
request correlation id '4850e0c7-5527-43e0-6bc-8d1a90fcd175'.","details":[{"target":"Microsoft.Compute/availabilitySets",
"message":"{\"error\":{\"details\":[{\"code\":\"MissingMoveDependentResources\",
\"target\":\"/subscriptions/7bce381f-79b3-4521-b36e-138932735300/resourceGroups/puzzlesmove-Migrated/providers/Microsoft.KeyVault/vaults
/puzzlesmove\",\"message\":\"The move resources request does not contain all the dependent resources.
Please check error details for missing resource ids.\"}],\"code\":\"MissingMoveDependentResources\",\"message\":
\"The move resources request does not contain all the dependent resources. Please check error details for missing resource ids.\"}}"}]}}
At line:1 char:1
+ Move-AzureRmResource -DestinationResourceGroupName move-subscription ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : CloseError: (:) [Move-AzureRmResource], ErrorResponseMessageException
+ FullyQualifiedErrorId : Conflict,Microsoft.Azure.Commands.ResourceManager.Cmdlets.Implementation.MoveAzureResourceCommand

Thinking I was smart, I instead went back and tried from the start again: ASM
Deploying a new vnet, storage account, and VM, I got a certificate on my Cloud Service, which I tried to delete. No cookie, we can’t even migrate to ARM…

vnet_no_cert

Testing with exact same setup, but not deleting my certificate:

vnet_cert

The good news is: Microsoft is aware of the Key Vault dependency issue, and they’re working on a fix. I hope to have it available very soon! And I guess the technician who got my case in Premier Support, is also hoping for a quick fix ;-)

17 Responses to ASM to ARM migration and fun with Key Vault

  1.  

    If customer is fully ARM (no cloud service) then migration between subscription is straightforward?

    • Short answer is yes.

      Long answer: It depends, of course the services they use, need to be on the list of supported services. In case they configured WinRM and use certificates stored in Key Vault, they will run into the same issue as above.

      I have a post in the making, that describes the flow to migrate between susbcriptions. May be ready tonight or during the weekend.

       
  2.  

    I’ve experienced the exact same problem! Hope there will be a fix soon.

    • There is a fix coming, but it’s unsure how fast we get it. It’s high priority on the teams list, but it needs testing and validation.

      I thought we had a workaround last week, but it still involves product group :(

       
    •  
  3.  

    Anynews for a fix? Got into the same “trap” with certificate dependency.

  4.  

    Any update on this? Waiting to switch my customer over to CSP but as I migrated all the way from classic I imagine this would be of some significant hindrance!

    • Fix is still being tested, that’s the latest update. I’m in contact with Product Group and will update here.

       
  5.  

    Thanks Jesper. I’m not convinced that this will happen before 2018. Having some experience with Azure support, they probably have cc’d in a whole bunch of irrelevant people and the emails are flying around between case managers with nobody doing the actual work.
    What other migration paths are there if downtime isn’t so much of an issue?

  6.  

    Hi Guys,

    I ran into this problem during an ASM -> ARM -> CSP migration last night and opened a Sev A with MS. This is the solution they provided that worked for us.

    This has to be done for every VM that had a Key Vault and does have the obvious drawback of breaking the WinRM which will need to be reconfigured.

    Hope this helps everyone

    $vmName = “VM-Name”
    $rgName = “Resource-Group”
    $vm = Get-AzureRmVM -ResourceGroupName $rgName -Name $vmName
    $vm.OSProfile.Secrets = New-Object -TypeName “System.Collections.Generic.List[Microsoft.Azure.Management.Compute.Models.VaultSecretGroup]”
    Update-AzureRmVM -ResourceGroupName $rgName -VM $vm

    • Hi Samuel,

      Thanks for posting!
      This was the same fix I got from support, but at that time they needed to involve Product Group to prepare resources before running the script.
      Just tested now, and it seems to work without involving support now. Will post an update shortly!

      /Jesper

       
  7.  
  8.  
  9.  

Let me hear your opinion

This site uses Akismet to reduce spam. Learn how your comment data is processed.