Efficiently export files to Microsoft Azure using cURL
Exporting files to Microsoft Azure Storage using cURL provides a flexible and efficient way to interact with cloud storage directly from the command line. This guide demonstrates how to export files to Azure Blob Storage using cURL, highlighting the recommended use of Microsoft Entra ID authentication, secure token management, and robust error handling practices.
The shell examples use Bash and cURL 7.76.0 or later for --fail-with-body.
Setting up Azure storage account
Before using cURL with Azure Storage, create an Azure Storage account and container through the Azure Portal or Azure CLI. We recommend enabling Microsoft Entra ID (formerly Azure AD) authentication for superior security and streamlined token management.
Authentication methods
Azure Blob Storage supports two primary authentication methods:
Microsoft Entra ID (recommended)
Obtain an access token using the Azure CLI. Ensure you have logged in using az login and have a
data-plane role such as Storage Blob Data Contributor scoped to the target container.
token=$(az account get-access-token --resource https://storage.azure.com/ --query accessToken -o tsv) || exit 1
: "${token:?Azure returned no access token}"
Use the token in cURL requests:
curl --fail-with-body --show-error -X PUT \
-H "Authorization: Bearer $token" \
-H "x-ms-version: 2023-11-03" \
-H "x-ms-blob-type: BlockBlob" \
-H "Content-Type: application/octet-stream" \
--upload-file "localfile.txt" \
"https://youraccount.blob.core.windows.net/container/remotefile.txt"
Shared Access Signature (alternative)
For scenarios where Microsoft Entra ID is not feasible, use SAS tokens with strict security
controls. This method also supports additional integrity checks using the Content-MD5 header:
set -o pipefail
content_md5=$(openssl dgst -md5 -binary localfile.txt | base64) || exit 1
curl --fail-with-body --show-error -X PUT \
-H "x-ms-blob-type: BlockBlob" \
-H "x-ms-version: 2023-11-03" \
-H "Content-Type: application/octet-stream" \
-H "Content-MD5: $content_md5" \
--upload-file "localfile.txt" \
"https://youraccount.blob.core.windows.net/container/remotefile.txt?your_sas_token"
Handling large file uploads
For files that exceed the limit of 5,000 MiB per write operation, use the Block Blob API to upload large files in chunks. With API versions 2019-12-12 and later, the maximum block size is 4,000 MiB, allowing a maximum blob size of nearly 190.7 TiB when using up to 50,000 blocks.
#!/bin/bash
set -euo pipefail
: "${token:?Acquire an Azure access token first}"
file="largefile.txt"
block_size=$((4000*1024*1024)) # 4,000 MiB blocks
base_url="https://youraccount.blob.core.windows.net/container/largefile.txt"
[ -s "$file" ] || { echo 'Use a single PUT for an empty file.' >&2; exit 1; }
temp_dir=$(mktemp -d)
trap 'rm -rf -- "$temp_dir"' EXIT
trap 'exit 130' INT
trap 'exit 143' TERM
run_id=$(openssl rand -hex 8)
# Isolate this run's parts so retries cannot pick up stale files.
split -a 5 -b "$block_size" "$file" "$temp_dir/block_"
blocks=("$temp_dir"/block_*)
[ "${#blocks[@]}" -le 50000 ] || { echo 'Too many blocks.' >&2; exit 1; }
printf '%s\n' '<?xml version="1.0" encoding="utf-8"?><BlockList>' > "$temp_dir/blocklist.xml"
index=0
# Upload blocks
for block in "${blocks[@]}"; do
block_id=$(printf '%s%08d' "$run_id" "$index" | base64 | tr -d '\n')
encoded_id=${block_id//+/%2B}
encoded_id=${encoded_id//\//%2F}
encoded_id=${encoded_id//=/%3D}
curl --fail-with-body --show-error --retry 3 --upload-file "$block" \
-H "Authorization: Bearer $token" \
-H "x-ms-version: 2023-11-03" \
"$base_url?comp=block&blockid=$encoded_id"
printf '<Latest>%s</Latest>\n' "$block_id" >> "$temp_dir/blocklist.xml"
index=$((index + 1))
done
printf '%s\n' '</BlockList>' >> "$temp_dir/blocklist.xml"
# Commit only after every block succeeded.
curl --fail-with-body --show-error --upload-file "$temp_dir/blocklist.xml" \
-H "Authorization: Bearer $token" \
-H "x-ms-version: 2023-11-03" \
-H "Content-Type: application/xml" \
"$base_url?comp=blocklist"
--upload-file streams each PUT; --data-binary @file would first load it into memory. This
example needs temporary disk space approximately equal to the input size. Keep the input unchanged
during the run. Block IDs must have equal lengths and be URL-encoded in the query string; the XML
list uses their original Base64 form. Failed runs remove their local parts and never commit a
partial list. Blob names containing spaces or reserved characters also need URL encoding.
Error handling and retries
Implement robust error handling with exponential backoff for transient errors. The function below retries single-file PUTs on HTTP 408, 429, 500, 502, 503, and 504, while immediately aborting on authentication failures (HTTP 401 or 403). It reports other transport or HTTP failures as failures:
upload_with_retry() {
local url="$1"
local file="$2"
local max_attempts=5
local attempt=1
local wait_time=2
local status
local curl_status
while [ $attempt -le $max_attempts ]; do
curl_status=0
status=$(curl --fail-with-body --silent --show-error -o /dev/null -w "%{http_code}" \
-H "Authorization: Bearer $token" \
-H "x-ms-version: 2023-11-03" \
-H "x-ms-blob-type: BlockBlob" \
--upload-file "$file" \
"$url") || curl_status=$?
if [ "$curl_status" -eq 0 ] && [ "$status" = 201 ]; then
return 0
elif [ "$status" = 401 ] || [ "$status" = 403 ]; then
echo "Authentication failed. Check credentials."
return 1
elif [ "$status" = 408 ] || [ "$status" = 429 ] || [ "$status" = 500 ] || [ "$status" = 502 ] || [ "$status" = 503 ] || [ "$status" = 504 ]; then
echo "Transient error $status encountered. Retrying in $wait_time seconds..."
else
echo "Upload failed with HTTP status $status and curl exit code $curl_status."
return 1
fi
attempt=$((attempt + 1))
[ "$attempt" -le "$max_attempts" ] || break
sleep $wait_time
wait_time=$((wait_time * 2))
done
echo "File upload failed after $max_attempts attempts."
return 1
}
Monitoring and validation
Check the upload status and blob metadata using the following command:
curl --fail-with-body --silent --show-error --head \
-H "Authorization: Bearer $token" \
-H "x-ms-version: 2023-11-03" \
"https://youraccount.blob.core.windows.net/container/file.txt"
Monitor transfer progress with the progress bar option:
curl --fail-with-body --show-error --progress-bar \
-H "Authorization: Bearer $token" \
-H "x-ms-version: 2023-11-03" \
-H "x-ms-blob-type: BlockBlob" \
--upload-file "largefile.txt" \
"https://youraccount.blob.core.windows.net/container/largefile.txt"
Security best practices
- Use Microsoft Entra ID authentication whenever possible to leverage improved security and token management.
- Always use HTTPS for secure data transfers with Azure Storage.
- Restrict permissions granted by SAS tokens and use short expiration times.
- Enforce IP address restrictions to limit access to your storage account.
- Set the minimum required permissions for each operation.
- Monitor and log all operations for audit and troubleshooting purposes.
- Regularly rotate credentials to mitigate potential security breaches.
- Validate file integrity using the
Content-MD5header.
Performance optimization
To maximize transfer performance:
- Choose appropriate block sizes (up to 4,000 MiB per block).
- Enable concurrent uploads where possible for multiple files.
- Select the closest Azure region to reduce latency.
- Monitor network bandwidth and latency to optimize transfers.
- Use compression when appropriate, considering that some files are already compressed.
- Implement comprehensive error handling and retries for robust uploads.
Conclusion
Integrating Azure Blob Storage with cURL offers a powerful strategy for managing cloud storage operations. By following these updated authentication methods, security best practices, and performance optimization tips, you can build a reliable and secure file transfer solution. For additional advanced file handling capabilities, consider exploring Transloadit's services, which support features like Uppy and tus for enhanced file processing.