Stream tar archives between servers without local storage

Kevin van Zonneveld

Co-founder · Amsterdam, The Netherlands ·

To copy a directory between two servers, pipe a source-side tar command through SSH into a destination-side tar command. The archive passes through your computer without being saved as a local file. The destination needs space for the extracted files, but neither server needs space for an intermediate archive.

The route is source server → your computer → destination server. Your computer carries the whole stream and must stay connected; the servers do not need SSH access to each other. If the bytes must bypass your computer entirely, run the transfer on a host with the required server connectivity and credentials.

SSH encrypts communication with each server. The archive is decrypted in the local pipe and encrypted again for the destination, so use a trusted computer. There is no need to add an OpenSSL password to this pipeline for transport encryption.

Prepare the source and destination

This example copies app-data in the source account’s home directory to a new app-data.incoming directory in the destination account’s home directory. Replace source-server and destination-server with your SSH aliases or user@host addresses. Adjust the paths in both scripts if your data lives elsewhere.

You need Bash on the computer running the scripts, and OpenSSH access to two Linux accounts with GNU tar, GNU find, and GNU sha256sum. The source account must be able to read every file; the destination account must be able to create the incoming directory and its contents. Configure key-based authentication, load any encrypted key into your SSH agent, and verify both servers’ host keys before starting. The commands use BatchMode=yes, so they fail instead of prompting for passwords or host-key confirmation during a transfer.

Stop writes to the source directory until copying and verification finish, or read from a consistent snapshot. For a web application, that includes uploads and background jobs. Use the database’s backup procedure for its data: copying live database files with tar does not establish a consistent database backup. These commands copy files; application setup and traffic cutover are separate steps.

The examples were tested with Bash 5.1.16, GNU tar 1.34, and OpenSSH 8.9p1 on Linux. The checksum step uses GNU find and coreutils; it is not a macOS command recipe.

Stream into a new directory

Save this as transfer.sh on your computer and run it with bash transfer.sh:

#!/usr/bin/env bash
set -euo pipefail

ssh -T -o BatchMode=yes source-server 'test -d app-data'
ssh -T -o BatchMode=yes destination-server 'mkdir app-data.incoming'

ssh -T -o BatchMode=yes source-server 'tar -cf - -C app-data .' |
  ssh -T -o BatchMode=yes destination-server 'tar -xf - -C app-data.incoming'

printf 'Transfer completed; verify app-data.incoming before using it.\n'

The plain mkdir deliberately fails if the incoming directory already exists. Choose a new path for a retry, or inspect and remove a failed transfer’s directory first. This keeps the example from extracting over an existing application directory.

Understanding tar streaming basics

In tar -cf -, c creates an archive and f - sends it to standard output. In tar -xf -, x extracts the archive read from standard input. These are the same operations commonly written as tar cf - and tar xf -.

-C app-data changes tar’s working directory before it archives .. Members therefore have names such as ./uploads/photo.jpg, including dotfiles, rather than a directory prefix you need to remove later. Extraction’s -C places those members under app-data.incoming. It does not create that directory.

-T disables SSH’s pseudo-terminal allocation so the connection can carry binary data. pipefail makes the pipeline fail if either SSH command fails; set -e then stops the script before its completion message. Without pipefail, a source tar process can fail after producing an extractable archive while the destination exits successfully. SSH returns the remote command’s exit status, which lets Bash detect that failure.

A successful run prints the completion message and leaves the copied tree in app-data.incoming. A failed run can leave partial files there. Neither a zero exit status nor the completion message proves that a changing source was copied consistently.

Verify the copied files

Keep the source unchanged and save this as verify.sh. Run bash verify.sh after the transfer:

#!/usr/bin/env bash
set -euo pipefail

ssh -T -o BatchMode=yes source-server \
  'cd app-data && find . -type f -exec sha256sum -- {} +' |
  ssh -T -o BatchMode=yes destination-server \
    'cd app-data.incoming && sha256sum --check --strict -'

printf 'Regular-file checksums match.\n'

This streams a checksum list between the servers, again without saving it locally. GNU sha256sum --check prints ./filename: OK for each matching file and fails for missing or mismatched files. GNU find -exec … {} + also returns failure if a checksum command fails, so the source side of this pipeline is checked. Expect the final message only when both sides succeed.

The check reads every regular file again on both servers. It assumes at least one regular file and verifies file contents, not symlink targets, empty directories, permissions, ownership, or extra destination files. This is a directory-copy recipe, not a full system backup: it does not request preservation of ACLs or extended attributes. Confirm the metadata your application needs before switching it to the copied directory.

Adjust the transfer when needed

For a progress display, install pv on your computer and insert it into the transfer pipeline: ssh … | pv | ssh …. Keep the pipeline inside the script with pipefail. The pv manual describes its byte count, elapsed time, and transfer rate. With no known total stream size, it cannot give a meaningful completion percentage or ETA. Its output measures bytes passing through the local pipe, not verified files.

If access requires a bastion host, add -J bastion-host to each SSH command in both scripts. Set up authentication and host-key verification for that host too. ProxyJump connects to the target through the bastion; the pipe between the two SSH commands still runs on your computer. Jump-host identity and port settings belong in your SSH configuration because command-line options generally apply to the final target.

For compressible data over a constrained connection, change tar -cf - to tar -czf - and tar -xf - to tar -xzf -. This uses tar’s --gzip option and requires gzip on both servers. It costs CPU time and may do little for already compressed video, images, or archives. Measure with your files and connection before assuming it will be faster.

Troubleshooting common issues

Permission denied or disk full: Read stderr from both SSH commands. Check source readability, destination permissions, free space, and available inodes. Fix the cause and start with a fresh incoming directory; do not use a partial copy.
“File changed as we read it”: Stop the writer or use a snapshot, then repeat the transfer and checksum check. Suppressing the warning cannot make a live copy consistent.
Unexpected archive errors: Remote shell startup files must not print greetings to stdout for noninteractive commands. That output would enter the archive stream. Keep diagnostics on stderr; do not merge stderr into the pipe with 2>&1 or |&.
Dropped connection: This tar stream has no resume operation. tmux or screen can keep a process running when you detach from its terminal, but cannot repair a broken SSH connection in the transfer pipeline. Restart into a fresh directory and verify it before use.