Efficient CLI file uploads with open-source tools

Tim Koschützki

Co-founder · Berlin, Germany · Show bio

Uploading files from a command line interface (CLI) keeps hands on keyboards and fits neatly into automation scripts, CI/CD pipelines, and headless server management. This post surveys some popular open source tools, shows how to set them up, and shares practical tips for reliability and security when you need to handle uploads on the CLI.

Why upload from the CLI?

There are several advantages to handling file uploads via the command line:

• Automation: Integrate file transfers seamlessly into build or deployment pipelines.
• Scheduling: Easily schedule tasks like off-peak backups using cron or systemd timers.
• Scripting: Combine CLI tools within shell scripts for complex, reproducible workflows.
• Remote Access: Avoid GUI friction when working over SSH on remote servers.

Compare popular CLI upload tools

Several excellent open-source tools cater to different CLI file upload needs:

The rest of this guide focuses on two common choices: s3cmd for robust cloud object storage uploads and transfer.sh for quick, ad-hoc file sharing. The principles apply broadly.

Use s3cmd for cloud storage uploads

s3cmd is a mature, feature-rich Python utility designed for interacting with Amazon S3 and S3-compatible object storage services like MinIO, Wasabi, Backblaze B2, and others.

Install s3cmd

Use a maintained Python 3 runtime. With pipx⁠ installed, keep s3cmd isolated from your system Python packages:

pipx install s3cmd

# Or use system package managers (May lag behind the latest version)
# Debian / Ubuntu
sudo apt-get update && sudo apt-get install s3cmd

# macOS via homebrew
brew install s3cmd

Verify the installation: s3cmd --version should display 2.4.0 or newer.

Configure credentials

Run the interactive configuration wizard:

s3cmd --configure

You'll be prompted for your Access Key ID, Secret Access Key, default region, and preferences for encryption and HTTPS. The settings are saved to a .s3cfg file in your home directory. For automated environments, consider using environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_SESSION_TOKEN for temporary credentials) or IAM roles instead of storing keys in the configuration file. Set the bucket region with --region or bucket_location in .s3cfg; AWS_DEFAULT_REGION does not configure s3cmd.

Upload examples

# Define your target bucket (replace with your actual bucket name)
aws_bucket="s3://your-unique-bucket-name"

# Upload a single file to a specific path within the bucket
s3cmd put report.pdf "$aws_bucket/reports/"

# Synchronize a local directory with a path in the bucket (additive: nothing is deleted)
s3cmd sync ./local-data/ "$aws_bucket/data-backup/"

# Only if you really want the remote to mirror the local side, including deletions.
# --delete-removed deletes every remote object with no local counterpart, so a typo
# in the source path wipes the prefix. Always confirm with --dry-run first.
s3cmd sync --delete-removed --dry-run ./local-data/ "$aws_bucket/data-backup/"

# Upload a file with S3 server-side encryption (SSE-S3)
# Note: --encrypt is *client-side* GPG encryption, which is a different thing
s3cmd put --server-side-encryption sensitive-data.zip "$aws_bucket/private/"

Tune for large files

For large file uploads, s3cmd supports multipart uploads, which break the file into smaller chunks.

# Upload a large file in 50 MiB chunks, with up to 5 retries and a progress bar
s3cmd put \
  --multipart-chunk-size-mb=50 \
  --max-retries=5 \
  --progress \
  large-video.mp4 "$aws_bucket/videos/"

Use transfer.sh for quick file sharing

transfer.sh provides a simple way to share files quickly from the command line. While public instances have existed, the project now primarily recommends self-hosting for reliability.

Run your own transfer.sh instance

You can easily run transfer.sh using Docker. The following command starts a temporary instance storing files in /tmp inside the container:

# Start a disposable server on host port 8080
# Files will be stored in /tmp inside the container
docker run -d --rm -p 127.0.0.1:8080:8080 \
  --name transfersh \
  dutchcoders/transfer.sh:latest \
  --provider local --basedir /tmp/

This disposable example is anonymous and bound to loopback only. Use non-sensitive test files; configure authentication and HTTPS before exposing an instance to other users. For persistent storage, mount a host directory to /tmp inside the container (e.g., -v /path/on/host:/tmp). Consult the transfer.sh documentation for more advanced configurations.

Share a file via your instance

Once your instance is running (replace localhost:8080 if needed):

# Upload diagram.png to your self-hosted transfer.sh
# The command outputs the shareable URL upon successful upload
curl --fail --show-error --upload-file ./diagram.png http://localhost:8080/diagram.png

Handy public alternatives

If self-hosting isn't practical for a quick, one-off share, several public services offer similar functionality:

# Upload to file.io (file expires after first download)
# -F builds a multipart/form-data body; -f makes curl fail on HTTP errors
curl -fsS -F "file=@backup.tar.gz" https://file.io

# Upload using temp.sh's documented multipart endpoint
curl -fsS -F "file=@notes.txt" https://temp.sh/upload

Check the current retention limits and terms on file.io⁠ and temp.sh⁠ before use. Do not send confidential files to an anonymous public service; an unguessable download URL is not access control.

Secure your CLI uploads

When automating file transfers, security is paramount:

1. Use HTTPS: Always prefer HTTPS endpoints (s3cmd uses HTTPS by default) to protect credentials and data in transit. Avoid plain HTTP.
2. Encrypt Sensitive Data: For server-side encryption, use s3cmd --server-side-encryption (SSE-S3) or --server-side-encryption-kms-id KEY_ID (SSE-KMS); the object is encrypted by the provider after it arrives. s3cmd --encrypt is the opposite: it is client-side GPG encryption, and it fails unless gpg_command and gpg_passphrase are set in .s3cfg. Standalone tools like age or gpg cover the same client-side ground.
3. Manage Credentials Securely: Avoid hardcoding access keys or secrets in scripts. Use environment variables, dedicated secrets management tools (like HashiCorp Vault), or IAM roles (for cloud environments like AWS EC2) that grant temporary, least-privilege credentials. Keep .s3cfg files out of version control.
4. Apply Least Privilege: Configure IAM policies or bucket policies to grant only the necessary permissions (e.g., s3:PutObject for uploads, but not s3:DeleteObject if deletion isn't required). Restrict public access unless absolutely necessary. Consider bucket policies that enforce encryption on upload.
5. Rate Limit: If performing bulk uploads, use rate-limiting options (s3cmd --limit-rate=1M for 1 MiB/s, rclone --bwlimit 1M) to prevent saturating network links or hitting API rate limits. Implement exponential backoff in scripts when encountering rate-limiting errors (like HTTP 429).

Automate with systemd timers

On modern Linux systems, systemd timers offer a robust alternative to cron for scheduling tasks. They handle missed runs (e.g., if the server was down) and provide better logging integration.

Here's how to set up a daily backup upload using s3cmd:

1. Create the service unit file /etc/systemd/system/backup-upload.service:

[Unit]
Description=Tar and upload nightly backup to S3
# Ensures network is up before starting
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
# Path to your backup script
ExecStart=/opt/scripts/backup-upload.sh
# Run the script as a specific user (create this user if needed)
User=backup
Group=backup
# Set environment variables for s3cmd if not using .s3cfg or IAM roles.
# These names are case-sensitive: lowercase variants are ignored.
# Environment="AWS_ACCESS_KEY_ID=your_key_id"
# Environment="AWS_SECRET_ACCESS_KEY=your_secret_key"
# Configure bucket_location in the backup user's .s3cfg for the bucket's region.

[Install]
WantedBy=multi-user.target

2. Create the timer unit file /etc/systemd/system/backup-upload.timer:

[Unit]
Description=Run backup-upload.service daily at 2 AM

[Timer]
# Run daily at 2:00 am
OnCalendar=*-*-* 02:00:00
# Run on boot if the last scheduled run was missed
Persistent=true
Unit=backup-upload.service

[Install]
WantedBy=timers.target

3. Create the backup script /opt/scripts/backup-upload.sh (ensure it's executable: chmod +x /opt/scripts/backup-upload.sh and owned by the backup user: chown backup:backup /opt/scripts/backup-upload.sh):

#!/usr/bin/env bash
# Exit immediately if a command exits with a non-zero status.
# Treat unset variables as an error.
# Pipelines fail if any command fails, not just the last one.
set -euo pipefail

# Configuration
S3_BUCKET="s3://your-unique-bucket-name/backups" # Replace with your bucket path
BACKUP_SOURCE_DIR="/var/www/my-app-data"         # Replace with the directory to back up
TIMESTAMP=$(date +%Y-%m-%d_%H-%M-%S)
# A private random directory prevents collisions and predictable /tmp symlink writes.
ARCHIVE_DIR=$(mktemp -d "${TMPDIR:-/tmp}/backup.XXXXXX")
ARCHIVE_FILE="$ARCHIVE_DIR/backup-${TIMESTAMP}.tar.gz"
trap 'rm -rf -- "$ARCHIVE_DIR"' EXIT
LOG_FILE="/var/log/backup-upload.log" # Ensure 'backup' user can write here

# Redirect stdout and stderr to the log file
exec >>"$LOG_FILE" 2>&1

echo "----------------------------------------"
echo "[$(date)] Starting backup process..."

# Create the compressed archive
echo "[$(date)] Creating archive: $ARCHIVE_FILE from $BACKUP_SOURCE_DIR"
# Use -C to change directory, avoiding leading paths in the archive
tar -czf "$ARCHIVE_FILE" -C "$(dirname "$BACKUP_SOURCE_DIR")" "$(basename "$BACKUP_SOURCE_DIR")"
echo "[$(date)] Archive created successfully."

# Upload to S3 using s3cmd
echo "[$(date)] Uploading $ARCHIVE_FILE to $S3_BUCKET"
# A backslash only continues a line when it is the *last* character on it,
# so keep comments on their own lines above the flags they describe.
# --storage-class: Infrequent Access, for cost savings
# --acl-private:   keep the object private
# --progress:      record transfer progress in the log
s3cmd put \
  --storage-class=STANDARD_IA \
  --acl-private \
  --progress \
  "$ARCHIVE_FILE" "$S3_BUCKET/"
echo "[$(date)] Upload completed."

echo "[$(date)] Backup process finished successfully."
echo "----------------------------------------"

exit 0

4. Enable and start the timer:

# Reload systemd so it picks up the new unit files
sudo systemctl daemon-reload
# Enable the timer to start on boot
sudo systemctl enable backup-upload.timer
# Start the timer now (it fires according to OnCalendar)
sudo systemctl start backup-upload.timer
# Check the status
sudo systemctl status backup-upload.timer
sudo systemctl status backup-upload.service
# List active timers
sudo systemctl list-timers --all

Remember to configure log rotation (e.g., using logrotate) for /var/log/backup-upload.log.

Troubleshoot common issues

Bring in Transloadit for web & mobile uploads

While CLI tools excel in server-side and automation contexts, handling uploads directly from web browsers or mobile applications requires different solutions that manage network interruptions, provide progress feedback, and offer resumability.

This is where Transloadit's /upload/handle Robot comes in. It's designed specifically for robust client-side uploads.

{
  "steps": {
    ":original": {
      "robot": "/upload/handle",
      "result": true
    }
  }
}

You can integrate this Robot with front-end libraries like Uppy (our versatile file uploader) or any client supporting the tus resumable upload protocol⁠. This combination provides features like pausing/resuming uploads, automatic retries, and real-time progress updates, enhancing the user experience for client-facing applications.

Happy uploading!