Leverage Azure CLI to automate cloud file transfers efficiently

Tim Koschützki

Co-founder · Berlin, Germany ·

Azure CLI is a versatile tool that allows you to automate file transfers and manage Azure Storage efficiently. This guide provides practical examples and step-by-step instructions to streamline your cloud workflows.

Installation and setup

Install Azure CLI on your system. After installation, run the examples in Bash (on Windows, use WSL or Azure Cloud Shell). Account creation and storage incur Azure charges; use a dedicated test resource group and non-sensitive sample files.

On Windows

winget install -e --id Microsoft.AzureCLI

On macOS

brew install azure-cli

On Linux (Ubuntu/Debian)

# Install pre-requisites
sudo apt-get update
sudo apt-get install -y ca-certificates curl apt-transport-https lsb-release gnupg

# Download and install the Microsoft signing key
sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://packages.microsoft.com/keys/microsoft.asc |
    gpg --dearmor |
    sudo tee /etc/apt/keyrings/microsoft.gpg > /dev/null
sudo chmod go+r /etc/apt/keyrings/microsoft.gpg

# Add the Azure CLI software repository
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/microsoft.gpg] https://packages.microsoft.com/repos/azure-cli/ $(lsb_release -cs) main" |
    sudo tee /etc/apt/sources.list.d/azure-cli.list

# Update repository information and install the azure-cli package
sudo apt-get update
sudo apt-get install azure-cli

After installation, authenticate with Azure:

az login

For automation hosted in Azure, prefer an assigned managed identity. On the Azure host that has that identity, replace interactive login with:

az login --identity

Verify the installation and check the version:

az --version

Creating a storage account

Select the intended subscription with az account set --subscription YOUR_SUBSCRIPTION_ID. Set these variables in the same Bash session, choosing a globally unique storage account name containing 3-24 lowercase letters and digits:

export RESOURCE_GROUP="file-transfer-demo"
export STORAGE_ACCOUNT="youruniquestorageaccount"
export CONTAINER_NAME="transfers"
export SOURCE_DIR="/path/to/local/files"
location="eastus"

The following account allows authenticated HTTPS access from your workstation while denying anonymous blob access and Shared Key authentication. It does not make the network endpoint private. For a private-only account, first configure a reachable private endpoint and DNS; disabling public network access before that would block the later transfers. See the storage account CLI reference.

# Create resource group
az group create --name "$RESOURCE_GROUP" --location "$location"

# Create storage account with security settings
az storage account create \
    --name "$STORAGE_ACCOUNT" \
    --resource-group "$RESOURCE_GROUP" \
    --location "$location" \
    --kind StorageV2 \
    --sku Standard_LRS \
    --min-tls-version TLS1_2 \
    --allow-blob-public-access false \
    --allow-shared-key-access false \
    --public-network-access Enabled \
    --https-only true \
    --encryption-services blob

Using Microsoft Entra ID authentication

Creating an account does not grant blob data access. An administrator with role-assignment permissions must grant the interactive user the Storage Blob Data Contributor role on this account. For automation, assign the role to the managed identity's principal object ID instead, with principal type ServicePrincipal. Role assignments can take several minutes to propagate.

storage_account_id=$(az storage account show \
    --name "$STORAGE_ACCOUNT" --resource-group "$RESOURCE_GROUP" --query id --output tsv)
user_id=$(az ad signed-in-user show --query id --output tsv)

# For the interactive user logged in above
az role assignment create \
    --role "Storage Blob Data Contributor" \
    --assignee-object-id "$user_id" \
    --assignee-principal-type User \
    --scope "$storage_account_id"

Transferring files using Azure CLI

Uploading files

Upload files using Microsoft Entra ID authentication:

# Create a container
az storage container create \
    --name "$CONTAINER_NAME" \
    --account-name "$STORAGE_ACCOUNT" \
    --auth-mode login

# Upload a file
az storage blob upload \
    --container-name "$CONTAINER_NAME" \
    --file /path/to/local/file.txt \
    --name remote-file.txt \
    --account-name "$STORAGE_ACCOUNT" \
    --overwrite false \
    --auth-mode login

Downloading files

Retrieve files using Microsoft Entra ID authentication:

az storage blob download \
    --container-name "$CONTAINER_NAME" \
    --name remote-file.txt \
    --file /path/to/local/destination.txt \
    --account-name "$STORAGE_ACCOUNT" \
    --overwrite false \
    --auth-mode login

Automating batch file transfers with Azure CLI

Save this Bash script and run it after exporting the configuration above and creating the container. It includes hidden regular files, skips directories and symbolic links, and makes at most three attempts per file. It never replaces existing blobs and returns a nonzero status if any upload fails, including a name collision. Use a new container or unique names for a fresh batch. Retrying an uncertain response is not proof that the earlier request failed: inspect the remote object before deciding to replace it.

#!/bin/bash
set -euo pipefail
shopt -s nullglob dotglob

: "${SOURCE_DIR:?Set SOURCE_DIR}"
: "${CONTAINER_NAME:?Set CONTAINER_NAME}"
: "${STORAGE_ACCOUNT:?Set STORAGE_ACCOUNT}"

if [ ! -d "$SOURCE_DIR" ]; then
    printf 'Source directory does not exist\n' >&2
    exit 1
fi

failed=0
max_retries=3
for file in "$SOURCE_DIR"/*; do
    [ -f "$file" ] && [ ! -L "$file" ] || continue
    filename=${file##*/}
    retry_count=0

    while [ $retry_count -lt $max_retries ]; do
        if az storage blob upload \
            --container-name "$CONTAINER_NAME" \
            --file "$file" \
            --name "$filename" \
            --account-name "$STORAGE_ACCOUNT" \
            --auth-mode login \
            --overwrite false \
            --only-show-errors --output none; then
            printf 'Uploaded: %s\n' "$filename"
            break
        else
            retry_count=$((retry_count + 1))
            if [ $retry_count -lt $max_retries ]; then
                printf 'Retry %s for %s\n' "$retry_count" "$filename" >&2
                sleep 5
            else
                printf 'Failed to upload %s after %s attempts\n' "$filename" "$max_retries" >&2
                failed=1
            fi
        fi
    done
done
exit "$failed"

Optimizing transfer performance

For large batches, az storage copy delegates transfers to AzCopy. Follow the Azure CLI copy reference and allow its AzCopy setup when prompted. This example copies the directory contents recursively, without replacing existing objects:

az storage copy \
    --source "$SOURCE_DIR/*" \
    --destination "https://$STORAGE_ACCOUNT.blob.core.windows.net/$CONTAINER_NAME" \
    --recursive \
    --put-md5 \
    --auth-mode login \
    -- --overwrite=false

Managing file access with Azure storage

Generate a short-lived, read-only user delegation SAS token. It grants access to known blob names throughout this container, so use a dedicated container or a blob-scoped SAS when sharing a single file. Treat the output as a secret: do not put it in logs, source control or public URLs. Python 3 provides a portable UTC expiry calculation on macOS and Linux:

end_time=$(python3 -c 'from datetime import datetime, timedelta, timezone; print((datetime.now(timezone.utc) + timedelta(minutes=30)).strftime("%Y-%m-%dT%H:%MZ"))')

az storage container generate-sas \
    --name "$CONTAINER_NAME" \
    --account-name "$STORAGE_ACCOUNT" \
    --permissions r \
    --expiry "$end_time" \
    --auth-mode login \
    --as-user \
    --https-only --output tsv

Monitoring and logging

To retain request logs, use an existing Log Analytics workspace and an identity permitted to configure diagnostic settings. Set WORKSPACE_GROUP and WORKSPACE_NAME to that workspace's resource group and name. Log ingestion and retention incur charges. Configure the blob service resource, not only the parent storage account, as described in the Blob Storage monitoring guide:

: "${WORKSPACE_GROUP:?Set WORKSPACE_GROUP}"
: "${WORKSPACE_NAME:?Set WORKSPACE_NAME}"
log_analytics_workspace_id=$(az monitor log-analytics workspace show \
    --resource-group "$WORKSPACE_GROUP" --workspace-name "$WORKSPACE_NAME" \
    --query id --output tsv)
workspace_id=$(az monitor log-analytics workspace show \
    --resource-group "$WORKSPACE_GROUP" --workspace-name "$WORKSPACE_NAME" \
    --query customerId --output tsv)

az monitor diagnostic-settings create \
    --name "storage-diagnostics" \
    --resource "$storage_account_id/blobServices/default" \
    --logs '[{"category": "StorageRead","enabled": true},{"category": "StorageWrite","enabled": true}]' \
    --export-to-resource-specific true \
    --workspace "$log_analytics_workspace_id"

# List all blobs in a container
az storage blob list \
    --container-name "$CONTAINER_NAME" \
    --account-name "$STORAGE_ACCOUNT" \
    --auth-mode login \
    --output table

Best practices

Use managed identities: Implement Microsoft Entra ID managed identities for authentication instead of storage account keys.

Enable soft delete: Protect against accidental deletions.

az storage account blob-service-properties update \
    --delete-retention-days 7 \
    --enable-delete-retention true \
    --account-name "$STORAGE_ACCOUNT" \
    --resource-group "$RESOURCE_GROUP"

Use private endpoints where needed: Configure a reachable virtual network and private DNS, verify access from the transfer host, then restrict public access. Creating an endpoint alone does not disable the public endpoint. Follow the private endpoint setup.

Enable versioning: Maintain multiple versions of your files.

az storage account blob-service-properties update \
    --account-name "$STORAGE_ACCOUNT" \
    --resource-group "$RESOURCE_GROUP" \
    --enable-versioning true

Implement lifecycle management: Review retention and retrieval costs before applying a policy. Scope it to the intended blob prefix and validate it on test data; deletion rules remove files. Use the lifecycle policy guide.
Choose infrastructure encryption at account creation if required. It is not an account-update toggle. See infrastructure encryption.

Troubleshooting common issues

Check firewall and public-endpoint settings. This management-plane query does not prove data access; also run the authenticated blob listing below:

az storage account show \
    --name "$STORAGE_ACCOUNT" --resource-group "$RESOURCE_GROUP" \
    --query '{publicNetworkAccess:publicNetworkAccess,networkRuleSet:networkRuleSet}'

View operation logs after enabling the diagnostic setting above. The query command may prompt to install the Azure CLI Log Analytics extension:

az monitor log-analytics query \
    --workspace "$workspace_id" \
    --analytics-query "StorageBlobLogs | where TimeGenerated > ago(1h) | take 20"

Test storage account access:
```
az storage blob list \
    --container-name "$CONTAINER_NAME" --account-name "$STORAGE_ACCOUNT" \
    --auth-mode login --output table
```
A permission error can mean the data role has not propagated, the wrong identity is logged in, or the network path is blocked. az storage account check-name only checks name availability; it is not an access test.

Next steps

Set up your first automated file transfer.
Configure lifecycle management for your storage account.
Enable versioning for critical files.

Need to handle complex file processing workflows? Check out Transloadit for comprehensive file importing and exporting services.