Batch export files to Amazon S3 with curl and AWS CLI

Kevin van Zonneveld

Co-founder · Amsterdam, The Netherlands · Show bio

In many development workflows, exporting files directly to cloud storage like Amazon S3 is a common requirement. Automating this file export process reduces errors and accelerates your workflow. In this guide, we explain how to batch export files to Amazon S3 by generating pre-signed URLs using AWS CLI and uploading via curl in a Bash script, providing an open source approach.

Prerequisites

Before you begin, you need:

• AWS CLI version 2 installed and configured with the appropriate credentials.
• curl installed on your system.
• Node.js 24 or newer, which runs TypeScript files directly. The signer below needs it.
• A Bash shell environment.
• An existing S3 bucket with the necessary permissions.
• An IAM user or role that can perform S3 actions.

Install AWS CLI v2 (Linux):

# For x86_64 systems
curl -fsSL --retry 3 "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o awscliv2.zip

# For arm64 systems
curl -fsSL --retry 3 "https://awscli.amazonaws.com/awscli-exe-linux-aarch64.zip" -o awscliv2.zip

unzip awscliv2.zip
sudo ./aws/install
rm -rf awscliv2.zip aws/ # Clean up: aws/ is a directory, so -f alone will not remove it
aws --version  # Should output aws-cli/2.x.x ...

Configure AWS CLI. The signer in the next section reads the same credential file, so this one step covers both tools:

aws configure
# Enter:
# - AWS Access Key ID [None]: your_access_key
# - AWS Secret Access Key [None]: your_secret_key
# - Default region name [None]: your-region (for example, us-east-1)
# - Default output format [None]: json (or leave blank)

Install the two AWS SDK packages the signer imports:

mkdir s3-batch-export && cd s3-batch-export
npm init -y
npm pkg set type=module
npm install @aws-sdk/client-s3 @aws-sdk/s3-request-presigner

Understand the components

Amazon S3 is a scalable object store. A pre-signed URL grants time-limited permission to perform one specific kind of S3 request without exposing your AWS credentials to whoever performs it. The signature covers the HTTP method, the bucket and key, the expiry, and any headers you choose to sign, so a URL signed for a GET cannot be replayed as a PUT.

It is not a one-shot token. The same URL can be redeemed as often as you like until it expires, and each redemption overwrites the object at that key. Treat it as a capability with a deadline: hand it to one party, keep it out of logs and referrers, and keep --expires-in short.

That detail decides the tooling. The aws s3 presign command is documented as GET-only: "This allows anyone who receives the pre-signed URL to retrieve the S3 object with an HTTP GET request." It has no --method flag, so its output cannot authorize curl -T; S3 answers such an attempt with SignatureDoesNotMatch. To sign a PUT you need a signer that lets you name the operation, which is what @aws-sdk/s3-request-presigner⁠ does. AWS CLI still earns its place here for credential setup and verification, and curl still performs the transfer.

Provide the required IAM permissions

The signature itself is computed locally, so no permission is checked while the URL is being minted. The permissions that matter are the signer's own, because S3 evaluates the signing identity's policy when the pre-signed URL is redeemed. Attach a policy like this to the IAM user or role whose credentials you configured:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:PutObject"],
      "Resource": "arn:aws:s3:::your-bucket-name/*"
    }
  ]
}

Replace your-bucket-name with your actual bucket name.

Generate pre-signed PUT URLs with the AWS SDK

Save this as presign-put.ts. It prints one URL and exits, which makes it easy to call from a shell loop later:

// presign-put.ts - print a pre-signed URL that authorizes PUT, and only PUT.
import { parseArgs } from 'node:util'

import { PutObjectCommand, S3Client } from '@aws-sdk/client-s3'
import { getSignedUrl } from '@aws-sdk/s3-request-presigner'

const { values } = parseArgs({
  options: {
    bucket: { type: 'string' },
    key: { type: 'string' },
    region: { type: 'string' },
    'content-type': { type: 'string', default: 'application/octet-stream' },
    'expires-in': { type: 'string', default: '3600' },
  },
})

const { bucket, key, region } = values
if (!bucket || !key || !region) {
  throw new Error('Usage: node presign-put.ts --bucket B --key K --region R [--content-type T]')
}

const expiresIn = Number(values['expires-in'])
// SigV4 caps a pre-signed URL at 7 days. Reject longer values here rather than
// relying on the SDK's generic expiry error; fractional values are rejected too.
if (!Number.isInteger(expiresIn) || expiresIn < 1 || expiresIn > 604800) {
  throw new Error(`--expires-in must be between 1 and 604800 seconds, got: ${values['expires-in']}`)
}

const client = new S3Client({
  region,
  // Default 'WHEN_SUPPORTED' hoists a CRC32 of an *empty* body into the query
  // string. S3 enforces it against the bytes curl actually sends, so every
  // upload would fail. Pre-signed PUTs must opt out.
  requestChecksumCalculation: 'WHEN_REQUIRED',
})

const url = await getSignedUrl(
  client,
  new PutObjectCommand({ Bucket: bucket, Key: key, ContentType: values['content-type'] }),
  {
    expiresIn,
    // Pin Content-Type into the signature. S3 honors whatever Content-Type the
    // uploader sends either way; signing it is what stops the uploader from
    // sending a *different* one, since any mismatch now fails the signature.
    signableHeaders: new Set(['content-type']),
  },
)

console.log(url)

Run it:

node presign-put.ts \
  --bucket your-bucket-name \
  --key object-key \
  --region your-region \
  --content-type text/plain \
  --expires-in 3600

Replace your-bucket-name, object-key (the desired name of the file in S3), and your-region with your specific values. The URL that comes back carries the signature and its expiry in the query string; it contains no secret key, but anyone holding it can write that one object, repeatedly, until it expires, so keep it out of logs.

--expires-in is a ceiling, not a guarantee. If you sign with temporary credentials, from an IAM role, an SSO session or sts:AssumeRole, the URL stops working the moment those credentials expire, which is often well before the expiry you asked for.

Upload files with curl

Use the curl -T flag to specify the local file to upload. Adding --retry makes the upload more resilient to transient network issues. The Content-Type header is not optional here: it is one of the signed headers, so it must match the --content-type you passed to the signer. The subshell keeps strict error handling local to this example, even when pasted into an existing Bash session.

(
  set -euo pipefail

  # Determine the MIME type dynamically (works on Linux and macOS)
  CONTENT_TYPE=$(file -b --mime-type localfile.txt)

  # Sign for that exact type, then send that exact type
  URL=$(node presign-put.ts \
    --bucket your-bucket-name \
    --key localfile.txt \
    --region your-region \
    --content-type "$CONTENT_TYPE")

  if status=$(curl -fsS --retry 3 --retry-delay 2 \
    -T localfile.txt \
    -H "Content-Type: $CONTENT_TYPE" \
    -o /dev/null -w '%{http_code}' \
    "$URL") && [[ "$status" =~ ^2[0-9][0-9]$ ]]; then
    echo 'Upload completed.'
  else
    echo 'Upload failed: expected an HTTP 2xx response.' >&2
    exit 1
  fi
)

Replace localfile.txt with your file path and the bucket and region with your own values. -f makes curl exit non-zero on an HTTP error instead of printing S3's XML error body as if it were a success, and -sS keeps the progress meter quiet while still showing real errors. The explicit 2xx check also rejects redirects, which -f alone does not treat as errors. Do not add -L: following a redirect would replay the body against a URL the signature does not cover.

Automate batch file exports

Uploading dozens of files manually is tedious. The script below loops through a directory, signs a PUT for each file, uploads it with curl, and exits non-zero if any single file failed, so a cron job or CI step actually notices.

#!/usr/bin/env bash

set -euo pipefail # Exit on error, undefined variable, or pipe failure

BUCKET="your-bucket-name"
EXPIRE=3600                     # URL validity in seconds (1 hour)
FILES_DIR="/path/to/your/files" # Directory containing files to upload
REGION="your-region"            # Your S3 bucket region
PRESIGN="./presign-put.ts"      # The signer from the previous section

# --- pre-flight checks ---

if [[ ! -d "$FILES_DIR" ]]; then
  echo "Error: Directory '$FILES_DIR' does not exist." >&2
  exit 1
fi

if [[ ! -f "$PRESIGN" ]]; then
  echo "Error: Signer '$PRESIGN' not found. See the previous section." >&2
  exit 1
fi

if ! command -v aws &>/dev/null; then
  echo "Error: AWS CLI command not found. Please install AWS CLI v2." >&2
  echo "See: https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html" >&2
  exit 1
fi

# Verify the credentials the signer will pick up are present and valid.
if ! caller_arn=$(aws sts get-caller-identity --query Arn --output text 2>/dev/null); then
  echo "Error: AWS credentials are not configured properly or are invalid." >&2
  echo "Please run 'aws configure' or check your environment variables/IAM role." >&2
  exit 1
fi
echo "AWS credentials verified for: $caller_arn"

echo "Starting batch export from '$FILES_DIR' to bucket '$BUCKET' in region '$REGION'..."

# --- processing loop ---

failed=0
uploaded=0

# dotglob: a hidden file is an ordinary object to S3, and silently leaving
# .env.example behind is data loss. nullglob: an empty directory must not run
# the loop once with a literal '*'.
shopt -s dotglob nullglob

for file in "$FILES_DIR"/*; do
  filename=$(basename "$file")

  # Test for a symlink before -f, which follows them: a link inside FILES_DIR
  # would otherwise upload bytes from outside it, under a name from inside it.
  if [[ -L "$file" ]]; then
    echo "  Skipping '$filename': symbolic link."
    continue
  fi

  # Skip directories and anything else that is not a regular file.
  [[ -f "$file" ]] || continue

  echo "Processing '$filename'..."

  # A single PUT tops out at 5 GiB; larger objects need multipart upload.
  # stat -f%z works on macOS/BSD, stat -c%s works on Linux.
  file_size=$(stat -f%z "$file" 2>/dev/null || stat -c%s "$file")
  if ((file_size > 5368709120)); then
    echo "  Skipping '$filename': $((file_size / 1024 / 1024)) MiB exceeds the 5 GiB PUT limit." >&2
    echo "  Use 'aws s3 cp' instead, which switches to multipart automatically." >&2
    failed=$((failed + 1))
    continue
  fi

  content_type=$(file -b --mime-type "$file")

  echo "  Signing a PUT for '$filename' (Content-Type: $content_type)..."
  if ! url=$(node "$PRESIGN" \
    --bucket "$BUCKET" \
    --key "$filename" \
    --region "$REGION" \
    --content-type "$content_type" \
    --expires-in "$EXPIRE"); then
    echo "  Error: Failed to sign a URL for '$filename'." >&2
    failed=$((failed + 1))
    continue
  fi

  echo "  Uploading '$filename'..."
  # The Content-Type must match the signature. curl -f does not reject redirects;
  # require a completed 2xx upload without replaying a signed body at another URL.
  if status=$(curl -fsS --retry 3 --retry-delay 2 \
    -T "$file" \
    -H "Content-Type: $content_type" \
    -o /dev/null -w '%{http_code}' \
    "$url") && [[ "$status" =~ ^2[0-9][0-9]$ ]]; then
    echo "  Successfully uploaded '$filename'."
    uploaded=$((uploaded + 1))
  else
    echo "  Error: Failed to upload '$filename'." >&2
    failed=$((failed + 1))
  fi
done

echo "Batch export completed: $uploaded uploaded, $failed failed."

# Surface partial failure to the caller: 'set -e' cannot do this for us, because
# every failure above was deliberately caught so the loop could continue.
if ((failed > 0)); then
  exit 1
fi

Remember to replace your-bucket-name, /path/to/your/files, and your-region in the script. Make the script executable (chmod +x script_name.sh) before running it.

Three limitations worth stating plainly. The script uses the filename as the object key, so files with the same name in different subdirectories would collide. That is why it only reads the top level of FILES_DIR. curl --retry resends the whole file from byte zero, because pre-signed PUT has no resume: a 4 GiB upload that dies at 90% starts over. And an upload is an unconditional overwrite, so re-running the script replaces whatever already sits at those keys. Enable bucket versioning if you need the previous bytes back.

Apply security best practices

When working with cloud resources, security is paramount:

• Prefer IAM Roles: When running scripts on EC2 instances or other AWS services, use IAM roles for temporary credentials instead of long-lived access keys.
• Least Privilege: Grant only the s3:PutObject permission needed for this task, scoped to the specific bucket.
• Short URL Lifetimes: Keep the --expires-in value for pre-signed URLs as short as practically possible for the upload duration.
• Encryption: Enable server-side encryption (SSE-S3, SSE-KMS, or SSE-C) on your S3 bucket to protect data at rest.
• Monitoring: Computing a signature emits no CloudTrail event, so nothing records that a URL was minted. What you can observe is the redemption: enable S3 server access logging or CloudTrail data events for the bucket, which record the resulting PutObject calls. Because a URL stays valid for its whole lifetime, those logs are also where you would notice one being replayed.
• VPC Endpoints: If your script runs within a VPC, use S3 VPC endpoints to keep traffic within the AWS network, avoiding the public internet.

Troubleshoot common issues

• Signer fails before any upload: The signature is computed locally, but resolving credentials is not always local: the default provider chain may call STS for an SSO or role session, or the instance metadata service on EC2, and either can fail or time out. Missing credentials surface as a CredentialsProviderError. Confirm your setup with aws sts get-caller-identity.
• SignatureDoesNotMatch during the curl upload: The request differs from what was signed. The usual causes are a Content-Type that does not match --content-type, a -L redirect, or a URL that was requoted or shell-expanded. Always wrap the URL in double quotes: it contains &.
• Access Denied during the curl upload: Verify the signing identity's policy allows s3:PutObject on arn:aws:s3:::your-bucket-name/*, and check bucket policies, Block Public Access settings, or a KMS key policy that might deny the write.
• XAmzContentChecksumMismatch or BadDigest: The URL was signed with the SDK's default checksum mode, which pins a CRC32 of an empty body. Set requestChecksumCalculation to 'WHEN_REQUIRED' as shown above.
• Network Timeouts (curl errors): Increase --retry counts or add --connect-timeout / --max-time options to curl if dealing with slow networks. Check network connectivity and firewalls.
• File Too Large (EntityTooLarge error or script skip): For files over 5 GiB, the single PUT operation used by this curl method won't work. Use the AWS CLI's aws s3 cp command, which handles multipart uploads automatically for large files.
• Region Mismatch (AuthorizationHeaderMalformed): The --region you pass to the signer is baked into both the hostname and the credential scope, so it has to match the bucket's actual region. aws s3api get-bucket-location --bucket your-bucket-name will tell you which that is.
• Expired URL (AccessDenied or Request has expired): The pre-signed URL is only valid for the duration specified by --expires-in. Regenerate the URL if the upload takes longer than expected or is attempted after expiration.
• Wrong Content Type: If files don't behave as expected after download, double-check the Content-Type header set during the curl upload. Ensure file --mime-type is giving the correct type.

Need to upload something bigger than 5 GiB? Let AWS CLI handle the complexity of multipart chunking for you:

# AWS CLI handles multipart uploads automatically for large files
aws s3 cp /path/to/your/large_file.zip s3://your-bucket-name/

Conclusion

Pairing a small AWS SDK signer with curl for the transfer gives you a flexible, open source pipeline for batch-exporting files to Amazon S3, with no credentials in the transfer command itself. The signer is the part aws s3 presign cannot cover, since its URLs only ever authorize a GET. Everything else stays in AWS CLI: credential setup, verification, and the large-file fallback.

For more complex workflows involving file processing before or after the file export to S3, consider a managed service. Our 🤖 /s3/store Robot, for example, wraps a similar pattern, allowing you to export results from other processing Steps directly to S3 within a single Assembly.

Happy coding!