Doylestown Coder

Love CI/CD in Azure DevOps! But if you make this one innocent mistake while using ARM Templates, you can completely utterly ruin your day. ARM Templates are scoped to a Resource Group. I use ARM Template deployments for deploying Azure Data Factory to QA and Prod

When configuring an ARM Template deployment watch out for this setting called “Deployment Mode”. It has two options in the dropdown:

• Incremental
• Complete

The default is incremental.

See that little “Info” icon. You gotta read that fine print. You can’t screw this one up! Also tell everyone on your team about this.

If you accidentally switch this to “Complete” and then run your CI/CD pipeline, say GOOD BYE to all your resources in the resource group that this deployment is scoped too. Because as soon as you click deploy, the first thing this does is nuke EVERYTHING in that resource group.

How do you prevent this?

• Lock your resources cause sooner or later someone might screw this up. See my previous blog on locking resources.
• Use deployment credentials that use the “least privileges” to get the job done.
• Limit resources in the Resource Group that the deployment is scoped to.

My Microsoft rep has commented on how easy it is to accidentally do this or have a junior guy flip the setting thinking it was just resetting a resource and redeploying just the templates.

I have a rule on our team. When deploying via ARM Templates, we create resource groups that only have resources in it that group are deployed via those templates. For ADF Resources, we create one resource group per ADF resource. Seems like an over kill but much easier to recover from just in case someone makes this mistake.

So use “INCREMENTAL” and lock your resources down. Your family will certainly thank me! Cause if you screw this up and nuke your prod environment, you’ll probably spend days rebuilding it.

Security Tip

Did you notice something? Hmm, what happens if your CI/CD account gets hacked and you don’t have locks and you are not using least privileges. So you already made a few bad mistakes… Now what if you only have one or two resource groups for your prod environment. Hmm, now the hacker quickly creates a CI/CD deployment using an ARM Template that is scoped to those groups. More than likely this won’t set off any red flags. Then he deploys. Your day is officially RUINED!

Yet another reason to use MFA on your Azure Accounts.

Conclusion

Pay attention to that setting. Don’t forget about least privilege for your deployment credentials – just as important as least privileges for accessing your databases. And lock those resources.