Imagining that we have a lot of data that we need to copy from one Azure storage account to another, we would normally use:
Problems
Azure Storage Explorer has a great UI and is super friendly to use but it’s not as flexible as AzCopy. For example, it doesn’t allow copying folders or files from a file share to a blob container.
AzCopy is the CLI tool running behind Azure Storage Explorer. It can perform copy operation at large scale but requires several manual steps and has some drawbacks:
- SAS (shared access signature) tokens: we need to generate a SAS token with a read permission for the source storage account and another with a write permission for the target storage account as they’re required by AzCopy.
- A powerful machine: running AzCopy can be very CPU intensive depending on how many files that we need to copy over. For example, copying more than one million of files may exhaust an 8-core processor.
- Security: running AzCopy locally might not be the safest way. We never know if our machines have been infected with any kind of viruses or malware.
Solution
The azcopy-runbook project is created to help tackle these problems. It utilizes an Azure Automation runbook with a hybrid worker (virtual machine) to run AzCopy.
The benefits of this approach are everything is automated and we can safely run the AzCopy command as long as we want on a customizable virtual machine.
The Azure resources required for running AzCopy are deployed seamlessly via the CI workflow. To run AzCopy, we can simply invoke the AzCopy workflow with optional parameters about the source and storage accounts.
Conclusion
AzCopy in an Azure Automation runbook seems to be a solid choice for transferring big data securely between different Azure storage accounts. It’s simply secure, performant and provides extra flexibility for customizations.
