Automate Azure CLI file transfers and verify the bytes

Tim Koschützki

Co-founder · Berlin, Germany ·

Use az storage blob upload to send a local file to Azure Blob Storage, then download it and compare the bytes. The script below does that for one file or a directory of files. It refuses existing blob names and reports a batch as complete only when every selected file passes the comparison.

Installation and setup

This walkthrough uses Bash on Linux and Microsoft's Azure CLI container, tested with Azure CLI 2.91.0. You need a working Docker installation, permission to run containers, and a writable working directory. Keep the source files unchanged during the transfer. Transfers and storage incur Azure charges; start with small, non-sensitive files.

The container supplies Bash and Python, so the script does not need a host Azure CLI installation. Follow Microsoft's container guide for the image's installation details. Pull the tested image and check the CLI inside it:

docker pull mcr.microsoft.com/azure-cli:2.91.0-azurelinux3.0 &&
docker run --rm mcr.microsoft.com/azure-cli:2.91.0-azurelinux3.0 az version

Choose the destination and authorization

Use an existing storage account and private blob container that your administrator has authorized you to write to. For new accounts, use general-purpose v2, as described in Microsoft’s account overview. Private here means anonymous blob access is disabled. Your machine still needs a working network path to the account's Blob endpoint.

This example uses Shared Key authentication. Your administrator must explicitly permit it and supply the account name and key through your secret manager as the exported environment variables AZURE_STORAGE_ACCOUNT and AZURE_STORAGE_KEY. The key grants broad account access: keep it out of scripts, shell history, and logs. Run this only where that authorization is appropriate.

Microsoft recommends Microsoft Entra authorization when available. With --auth-mode login, the identity needs a blob data role; a storage management role alone is insufficient. This script deliberately requires a supplied key; it does not log in, look up account keys, create resources, or assign roles. An account that disables Shared Key needs a different authentication setup.

Transferring files using Azure CLI

Save the following as transfer.sh in your working directory. Its four arguments are the source file or directory, existing container name, new blob-name prefix, and new download directory. Use relative paths beneath that working directory, such as ./input.bin and ./downloads, so the container can reach them. The download directory's parent must already exist.

Source paths and filenames must be valid UTF-8 without carriage returns or newlines. Spaces, literal percent signs, #, ?, leading hyphens, and Unicode filenames are supported. The prefix uses lowercase ASCII letters, digits, slashes, underscores, and hyphens; start it with a letter or digit and avoid empty segments, . or ... A prefix such as transfers/run-001 creates nested blob names without needing to create remote directories.

#!/usr/bin/env bash
set -uo pipefail

for tool in az python3 mkdir sleep; do
    command -v "$tool" >/dev/null || { printf 'Missing tool: %s\n' "$tool" >&2; exit 1; }
done
: "${AZURE_STORAGE_ACCOUNT:?Set AZURE_STORAGE_ACCOUNT}"
: "${AZURE_STORAGE_KEY:?Set AZURE_STORAGE_KEY}"
unset AZURE_STORAGE_CONNECTION_STRING AZURE_STORAGE_SAS_TOKEN AZURE_STORAGE_SERVICE_ENDPOINT

if [ "$#" -ne 4 ]; then
    printf 'Usage: bash transfer.sh SOURCE CONTAINER PREFIX DOWNLOAD_DIR\n' >&2
    exit 2
fi
source_path=$1
container=$2
prefix=$3
download_dir=$4
case "$source_path" in /*) ;; *) source_path="./$source_path" ;; esac
case "$download_dir" in /*) ;; *) download_dir="./$download_dir" ;; esac

if ! python3 - "$source_path" "$prefix" "$download_dir" <<'PY'
import os
import re
import sys

try:
    source, prefix, destination = sys.argv[1:]
    paths = [source, destination]
    if os.path.isdir(source):
        with os.scandir(source) as entries:
            paths.extend(entry.path for entry in entries
                         if entry.is_file(follow_symlinks=False))
    for path in paths:
        os.fsencode(path).decode('utf-8')
        if '\n' in path or '\r' in path:
            raise ValueError()
    if not re.fullmatch(r'[a-z0-9][a-z0-9/_-]*', prefix):
        raise ValueError()
    if any(part in ('', '.', '..') for part in prefix.split('/')):
        raise ValueError()
except (OSError, UnicodeError, ValueError):
    print('Check source access, UTF-8 paths, and the blob prefix.', file=sys.stderr)
    sys.exit(1)
PY
then
    exit 1
fi

if [ -L "$source_path" ] || { [ ! -f "$source_path" ] && [ ! -d "$source_path" ]; }; then
    printf 'Source must be a regular file or directory, not a symbolic link.\n' >&2
    exit 1
fi
mkdir -- "$download_dir" || exit 1
shopt -s nullglob dotglob
files=("$source_path")
if [ -d "$source_path" ]; then files=("$source_path"/*); fi

failed=0
verified=0
max_retries=3
for file in "${files[@]}"; do
    [ -f "$file" ] && [ ! -L "$file" ] || continue
    filename=${file##*/}
    blob_name="$prefix/$filename"
    uploaded=0
    for ((attempt=1; attempt<=max_retries; attempt++)); do
        if az storage blob upload --account-name "$AZURE_STORAGE_ACCOUNT" \
            --container-name "$container" --name "$blob_name" --file "$file" \
            --type block --auth-mode key --overwrite false \
            --only-show-errors --output none >/dev/null 2>&1; then
            uploaded=1
            break
        fi
        if [ "$attempt" -lt "$max_retries" ]; then sleep 5; fi
    done
    if [ "$uploaded" -eq 0 ]; then
        printf 'Upload refused or failed: %s\n' "$filename" >&2
        failed=$((failed + 1))
        continue
    fi
    downloaded_file="$download_dir/$filename"
    if ! az storage blob download --account-name "$AZURE_STORAGE_ACCOUNT" \
        --container-name "$container" --name "$blob_name" --file "$downloaded_file" \
        --auth-mode key --overwrite false --only-show-errors --output none >/dev/null 2>&1; then
        printf 'Download failed: %s\n' "$filename" >&2
        failed=$((failed + 1))
        continue
    fi
    if ! python3 - "$file" "$downloaded_file" <<'PY'
import filecmp
import sys

sys.exit(0 if filecmp.cmp(sys.argv[1], sys.argv[2], shallow=False) else 1)
PY
    then
        printf 'Byte comparison failed: %s\n' "$filename" >&2
        failed=$((failed + 1))
        continue
    fi
    printf 'Verified: %s\n' "$filename"
    verified=$((verified + 1))
done
printf 'Verified %s file(s); failed %s.\n' "$verified" "$failed"
if [ "$failed" -ne 0 ]; then exit 1; fi
printf 'Transfer complete\n'

Uploading files

Place a file named input.bin beside the saved script. With the two credential variables already exported, replace your-container with your existing container and choose a fresh prefix. Run:

docker run --rm --user "$(id -u):$(id -g)" \
    --env HOME=/tmp --env AZURE_CONFIG_DIR=/tmp/azure \
    --env AZURE_CORE_COLLECT_TELEMETRY=no \
    --env AZURE_STORAGE_ACCOUNT --env AZURE_STORAGE_KEY \
    --mount "type=bind,source=$PWD,target=/work" --workdir /work \
    mcr.microsoft.com/azure-cli:2.91.0-azurelinux3.0 \
    bash transfer.sh ./input.bin your-container transfers/run-001 ./downloads

The container exits afterward; the uploaded blob remains in Azure and the download remains in your working directory. The --user option keeps the downloaded files owned by your Linux user.

Downloading files

The script's az storage blob download retrieves the exact blob name it uploaded, and Python compares the complete download against the local source. A successful single-file run prints Verified: input.bin, then Verified 1 file(s); failed 0. and Transfer complete. Empty files are valid too. A successful upload followed by a failed download or comparison returns a nonzero status and leaves the remote blob in place for inspection.

Both transfers explicitly use --overwrite false; see the upload reference and download reference. The script also refuses an existing download directory before contacting Azure. To repeat a run, choose a new prefix and new download directory. Reusing a blob name with a fresh download directory fails instead of replacing the earlier blob. That refusal is intentional; the script does not implement in-place replacement or rollback.

Automating batch file transfers with Azure CLI

Put your files in ./input, then use the same script with that directory as its first argument:

docker run --rm --user "$(id -u):$(id -g)" \
    --env HOME=/tmp --env AZURE_CONFIG_DIR=/tmp/azure \
    --env AZURE_CORE_COLLECT_TELEMETRY=no \
    --env AZURE_STORAGE_ACCOUNT --env AZURE_STORAGE_KEY \
    --mount "type=bind,source=$PWD,target=/work" --workdir /work \
    mcr.microsoft.com/azure-cli:2.91.0-azurelinux3.0 \
    bash transfer.sh ./input your-container transfers/batch-001 ./batch-downloads

This selects regular files immediately inside ./input, including hidden files. It skips subdirectories and symbolic links; it does not recurse. Each selected filename becomes the last segment of its blob name and its local download name. An empty directory completes with zero verified files, so check the printed count if you expected a larger inventory.

Transfers run sequentially. Each upload gets at most three CLI invocations, separated by five seconds after a failure; the CLI's own service retries are additional. Downloads get one CLI invocation. The final count includes all failed uploads, downloads, and comparisons, even if later files succeed. A partially successful batch exits nonzero and keeps its successful blobs and downloads. It never prints Transfer complete for that result.

Troubleshooting common issues

If a credential variable is missing, the script stops before creating the download directory. For an upload refusal, check the selected account, container, key authorization, and network path with the administrator. The script reports the affected filename and suppresses raw CLI diagnostics to keep credentials and signed URLs out of its output.

An upload failure can also mean the blob name already exists. Three refused attempts do not make replacement safe. A lost response can leave a committed blob even when the command failed; inspect that exact name before retrying. Use a fresh prefix for another copy instead of changing the overwrite policy to get past a refusal.

If downloading or comparing fails, inspect the retained blob and local download separately. Keep the original source intact. The script does not delete partial downloads, undo successful uploads, or change account retention settings. Arrange any cleanup with the resource owner using the exact blob names from that run.

Next steps

For direct HTTP requests rather than CLI automation, see exporting files to Azure with cURL. That is a separate REST workflow; its authentication and upload program are not required here.