Efficient CLI file uploads with open-source tools
An upload command returning a URL does not tell you whether the downloaded file matches your input.
This walkthrough uses curl and a disposable local transfer.sh server to upload one nonempty file,
download it, and compare the bytes. You finish with a verified copy on disk and a SHA-256 checksum,
without a cloud account.
Compare popular CLI upload tools
Choose the tool around the receiving system. curl supplies the HTTP client; transfer.sh supplies
the server in the example below.
| Tool | Use it for | Check before choosing |
|---|---|---|
s3cmd | Uploads and sync to S3 or compatible object storage | You need a bucket and credentials with permission for the intended operations. |
rclone | Copying and syncing across cloud storage backends | Checksums and other capabilities vary by backend. |
curl | HTTP uploads to an existing endpoint | The endpoint determines the method, authentication, and response format. |
rsync | Synchronizing files with a machine you control | A remote shell transfer needs rsync on both machines. |
lftp | FTP or SFTP transfers and mirroring | Choose a protocol and account supported by the destination. |
For a storage-backed workflow, see batch exports to S3 with signed PUT URLs or rclone with DigitalOcean Spaces. Those tasks require provider configuration. Here, the receiving server runs on your own machine and is removed after the round trip.
Use transfer.sh for quick file sharing
This is an anonymous HTTP test on loopback, using a local Linux Docker daemon. The URL is reachable from that machine while the container runs; it is not a public sharing link. Use a non-sensitive, unchanging test file. There is no persistent server storage, authentication, resumability, or speed benchmark in this example.
The pinned release rejects zero-byte uploads. The script below refuses them before creating a result directory or container. If empty files are part of your transfer job, choose a destination that supports them.
The commands were tested on Linux x86-64 with Bash 5.3.15, curl 8.22.0, Docker Engine 29.7.2,
GNU coreutils 9.11, and GNU diffutils 3.12. You need docker, bash, curl, cmp, and sha256sum
on your executable search path, plus access to the local Docker daemon. A remote Docker context would
put the server on another machine. Docker documents that releases before 28.0.0 can expose
localhost-published ports to hosts on the same network segment.
Pull the exact image used below:
docker pull dutchcoders/transfer.sh@sha256:9383e66489ab3a7a56bec1b67d2e27d41c072102d515cdc5ab35f913b72e8a09
This digest pins transfer.sh v1.6.1,
released on December 4, 2023. Treat it as a reproducible local exercise, rather than a recommendation
to deploy that image publicly. The project’s Docker instructions
explain why a versioned image is preferable to the moving latest tag.
Run your own transfer.sh instance
Save this as upload-check.sh in a scratch directory. Run it with Bash, rather than sourcing it into
your shell. It creates a new result directory, allocates a host port automatically, waits for its
own container to become ready, and removes that container before reporting success.
The local storage and temporary upload files share a 64 MiB memory-backed mount. Leave room for
metadata and temporary files; this is a small-file exercise, not a large-file service.
The server receives a fixed filename, payload.bin, so spaces, leading hyphens, and URL punctuation
in your local filename do not become URL syntax.
#!/usr/bin/env bash
set -euo pipefail
umask 077
fail() { printf '%s\n' "$*" >&2; exit 1; }
[[ $# -eq 2 ]] || fail 'Usage: bash upload-check.sh INPUT NEW_RESULT_DIRECTORY'
input=$1
result_dir=$2
# Prefix relative paths so a literal "-" is a file, not curl's stdin selector.
[[ $input == /* || $input == ./* || $input == ../* ]] || input=./$input
[[ $result_dir == /* || $result_dir == ./* || $result_dir == ../* ]] || result_dir=./$result_dir
[[ -f $input && -r $input ]] || fail "Input is not a readable regular file: $input"
[[ -s $input ]] || fail "transfer.sh v1.6.1 rejects empty uploads: $input"
port=${UPLOAD_PORT:-0}
[[ $port =~ ^[0-9]{1,5}$ ]] || fail 'UPLOAD_PORT must be 0 or a port from 1 to 65535.'
(( 10#$port <= 65535 )) || fail 'UPLOAD_PORT exceeds 65535.'
port=$((10#$port))
name=${UPLOAD_NAME:-cli-upload-${RANDOM}-${RANDOM}-$$}
image=dutchcoders/transfer.sh@sha256:9383e66489ab3a7a56bec1b67d2e27d41c072102d515cdc5ab35f913b72e8a09
# An existing result directory is never reused or removed.
mkdir -- "$result_dir" || fail "Choose a new result directory: $result_dir"
work=$result_dir/.work
mkdir -- "$work"
verified=0
checksum=''
cleanup() {
status=$?
trap - EXIT
# A CID file identifies only the container this invocation created.
if [[ -s $work/container.id ]]; then
container_id=$(< "$work/container.id")
if ! docker rm --force "$container_id" >/dev/null; then
printf 'Cleanup failed; remove container %s when Docker is available.\n' "$container_id" >&2
status=1
fi
fi
if (( status == 0 && verified == 1 )); then
mv -- "$work/download.part" "$result_dir/download.bin" || status=1
fi
rm -rf -- "$work" || status=1
if (( status == 0 && verified == 1 )); then
printf 'Verified: %s\nSHA-256: %s\n' "$result_dir/download.bin" "$checksum"
fi
exit "$status"
}
trap cleanup EXIT
trap 'exit 130' INT
trap 'exit 143' TERM
trap 'exit 129' HUP
docker create --cidfile "$work/container.id" --name "$name" --pull=never \
--publish "127.0.0.1:$port:8080" \
--read-only --user 5000:5000 --cap-drop ALL --security-opt no-new-privileges \
--tmpfs /tmp:rw,size=64m,mode=1777 --memory 128m --cpus 1 --pids-limit 64 \
"$image" --provider local --basedir /tmp/data --temp-path /tmp \
> "$work/create.log"
container_id=$(< "$work/container.id")
docker start "$container_id" >/dev/null
binding=$(docker port "$container_id" 8080/tcp)
[[ $binding == 127.0.0.1:* ]] || fail 'Expected a loopback port binding.'
origin=http://$binding
curl_local() {
curl --disable --noproxy '*' --globoff --fail --silent --show-error \
--connect-timeout 2 --max-time 30 "$@"
}
ready=0
for ((attempt=0; attempt<40; attempt++)); do
if [[ $(docker inspect --format '{{.State.Running}}' "$container_id") != true ]]; then
docker logs "$container_id" >&2
fail 'Server exited before readiness.'
fi
if [[ $(curl_local --max-time 1 --output /dev/null --write-out '%{http_code}' \
"$origin/health.html" 2>/dev/null) == 200 ]]; then
ready=1
break
fi
sleep 0.25
done
if (( ready == 0 )); then
docker logs "$container_id" >&2
fail 'Server did not become ready.'
fi
if ! upload_status=$(curl_local --upload-file "$input" --output "$work/url.txt" \
--write-out '%{http_code}' "$origin/payload.bin"); then
docker logs "$container_id" >&2
fail "Upload failed: $input"
fi
[[ $upload_status == 200 ]] || fail "Unexpected upload HTTP status: $upload_status"
url=$(< "$work/url.txt")
path=${url#"$origin/"}
[[ $url == "$origin/"* && $path =~ ^[A-Za-z0-9]+/payload\.bin$ ]] \
|| fail 'Server returned an unexpected download URL.'
if ! download_status=$(curl_local --output "$work/download.part" \
--write-out '%{http_code}' "$url"); then
fail 'Download failed.'
fi
[[ $download_status == 200 ]] || fail "Unexpected download HTTP status: $download_status"
cmp -- "$input" "$work/download.part" || fail "Downloaded bytes differ: $input"
checksum=$(sha256sum < "$work/download.part")
checksum=${checksum%% *}
verified=1
--upload-file makes curl send an HTTP PUT.
--fail makes HTTP errors fail the command, and the script also
requires HTTP 200 at each stage. --disable is curl’s
first option so a personal configuration file cannot add redirects or change the request.
--globoff keeps curl from expanding brackets and braces in local filenames. The URL returned by the
server must point back to the exact origin Docker allocated.
Finally, cmp checks every downloaded byte before the script keeps download.bin.
The script cleans up on normal exit, Ctrl+C, and TERM or HUP signals. A signal sent only to Bash can
wait for the active command to return; each curl transfer has a 30-second deadline. It removes
containers by their created ID, so a name collision cannot remove somebody else’s container. SIGKILL,
a machine crash, or loss of the Docker daemon can prevent cleanup. UPLOAD_NAME lets you choose a
recognizable name for a run; UPLOAD_PORT lets you request a particular host port instead of the
default automatic allocation. Neither option makes an existing container or occupied port reusable.
Share a file via your instance
Create a small binary fixture and run the saved script. The parentheses contain the shell options;
noclobber refuses to overwrite an existing sample.bin. Use a directory where that filename and
upload-result are both unused.
(
set -euo pipefail
set -o noclobber
printf '\000\377\001\200\012\015\052\000\101\102' > ./sample.bin
bash ./upload-check.sh ./sample.bin ./upload-result
)
Successful output for those ten bytes is:
Verified: ./upload-result/download.bin
SHA-256: d77823e7a78045d088fa69d1572861efee50fa35240f7555fa860d02d22633d5
Open upload-result/download.bin or compare it with sample.bin; it remains after the server is
removed. To upload your own file, replace ./sample.bin in the Bash invocation and choose a new
result directory. Keep the input unchanged until the command finishes.
A failure returns nonzero, removes temporary downloads, and leaves the newly created result directory
without a verified copy. A rerun with that same result directory is refused, even if it is empty.
Secure your CLI uploads
Loopback limits this example to the local host; other processes and users on that host can still reach its anonymous endpoint. The container’s stored files disappear when it is removed, but the original input and your verified download remain on disk. Plain HTTP here is a local testing choice. For a service reachable by other machines, choose authenticated HTTPS and a retention policy before sending real data. An unguessable URL does not authenticate the person downloading it.
Troubleshoot common issues
| Symptom | What to check |
|---|---|
| Docker cannot create or start the container | Confirm the pinned image was pulled and the local daemon is accessible. An occupied UPLOAD_PORT or existing UPLOAD_NAME fails startup; choose another value. |
| Server exits or never becomes ready | Read the container diagnostic printed before cleanup. The script does not upload until its own server passes the health check. |
| Upload or download fails | Read curl’s error and the stage message. The server’s 64 MiB temporary mount can fill; try a smaller file. Transfers have a 30-second timeout. |
| Downloaded bytes differ | Check whether the input changed during transfer. A successful HTTP request alone is insufficient; the script discards this download. |
| Result directory already exists | Choose a new directory. The script preserves an earlier result instead of replacing it. |
| You need bandwidth limits for a cloud job | Use --limit-rate with s3cmd, or --bwlimit with rclone; these flags belong to different tools. |
When using this check in automation, rely on its exit status and give each run a new result directory. Scheduling it does not turn the temporary container into a persistent backup or public file service.
Continue with browser uploads
For a browser upload interface, Uppy provides file selection and upload progress. The tus protocol addresses resumable HTTP uploads when the server supports it. Those are separate client/server choices; this transfer.sh PUT example does not implement tus. Use the local round trip to check your CLI workflow, then choose the receiving service around the storage and access requirements of your application.