Last updated: June 13, 2025
# Stream Tar archives between servers without local storage
**Kevin van Zonneveld**
Co-founder · Amsterdam, The Netherlands · Show bio
[](https://x.com/kvz)[](https://github.com/kvz)
Transferring files between servers can be challenging, especially when local storage is limited. Fortunately, you can bypass local storage entirely by streaming tar archives directly between servers using SSH tunnels. This method is particularly useful for server migrations, backups, and managing large datasets.
## Understanding Tar streaming basics
The `tar` command is versatile and can stream data directly through pipes. When creating a tar archive to be extracted elsewhere, it's crucial to change the directory to the source path first, or use the `-C` option for `tar cf`. Otherwise, the archive will contain the full path from the root of the source server, which is usually not desired on the destination server. The basic syntax for creating and extracting archives using pipes correctly is:
```bash
tar cf - -C /path/to/source . | tar xf - -C /path/to/destination
```
Here, `tar cf - -C /path/to/source .` creates an archive of the contents of `/path/to/source` (with paths relative to this directory using `.`) and streams it to stdout. Then,`tar xf - -C /path/to/destination` reads from stdin and extracts the archive into`/path/to/destination`. This ensures that the files are extracted into the intended directory structure.
## Direct server-to-server transfers using ssh
To transfer files directly between two remote servers, combine `tar` with SSH:
```bash
ssh user@source-server "tar cf - -C /path/to/source ." | ssh user@destination-server "tar xf - -C /path/to/destination"
```
This command streams the archive directly from the source server to the destination server without intermediate storage.
## Advanced techniques: ssh proxyjump
If your servers are behind a bastion host, use SSH's `ProxyJump` feature:
```bash
ssh -J user@bastion-host user@source-server "tar cf - -C /path/to/source ." | ssh -J user@bastion-host user@destination-server "tar xf - -C /path/to/destination"
```
This securely tunnels your tar stream through the bastion host.
## Monitoring progress with pipe viewer (pv)
For large transfers, monitoring progress is helpful. Use `pv` to visualize the transfer:
```bash
ssh user@source-server "tar cf - -C /path/to/source ." | pv | ssh user@destination-server "tar xf - -C /path/to/destination"
```
`pv` provides real-time feedback on transfer speed and progress.
## Encrypting streams on-the-fly
To secure your data during transfer, encrypt the stream. For production environments, using key files or prompting for a password (by omitting `-pass`) is generally better than hardcoding passwords. When using `openssl enc` for symmetric encryption, consider using a key derivation function like PBKDF2, enabled by the `-pbkdf2` option (available in modern OpenSSL versions) for enhanced security:
```bash
# Consider prompting for password or using key files for better security
ssh user@source-server "tar cf - -C /path/to/source . | openssl enc -aes-256-cbc -pbkdf2 -salt -pass pass:yourpassword" | ssh user@destination-server "openssl enc -d -aes-256-cbc -pbkdf2 -pass pass:yourpassword | tar xf - -C /path/to/destination"
```
Alternatively, use `gpg` for encryption. You can use any valid GPG key identifier (such as a key ID, fingerprint, or an email associated with the key) for the `--recipient`. The recipient must have the corresponding private key to decrypt the stream.
```bash
ssh user@source-server "tar cf - -C /path/to/source . | gpg --encrypt --recipient your-key-identifier" | ssh user@destination-server "gpg --decrypt | tar xf - -C /path/to/destination"
```
## Real-world example: migrating a web application
Imagine migrating a web application between cloud providers. You can stream the entire application directory directly:
```bash
ssh user@old-server "tar cf - -C /var/www/html ." | pv | ssh user@new-server "tar xf - -C /var/www/html"
```
This approach minimizes downtime and ensures a smooth transition.
## Performance considerations and optimization tips
* **Compression**: Use compression flags with `tar` to reduce bandwidth usage. For example, `z` for gzip, `j` for bzip2, or `J` for xz. The command below uses gzip (`z`):
```bash
ssh user@source-server "tar czf - -C /path/to/source ." | ssh user@destination-server "tar xzf - -C /path/to/destination"
```
To use bzip2, you would use `cjf` on the source and `xjf` on the destination. For xz, use `cJf` and`xJf` respectively. Choose the algorithm based on the desired balance between compression ratio and CPU usage (gzip is fast, xz offers higher compression).
* **Network Speed**: Ensure your network connection is stable and fast enough to handle large transfers.
* **Resource Management**: Monitor CPU and memory usage during transfers to avoid overloading your servers.
## Troubleshooting common issues
* **Permission Errors**: Ensure the user has appropriate read permissions on all source files and directories, and write permissions on the destination directory.
* **Network Interruptions**: Use tools like `screen` or `tmux` to maintain sessions during network interruptions. This allows the command to continue running in the background and lets you reattach to the session even if your local connection drops.
* **Corrupted Archives**: Verify integrity by checking file sizes. For more robust checks, compare checksums (e.g., using `md5sum` or `sha256sum`) of a few key files on both the source and destination after the transfer.
## Conclusion
Streaming tar archives directly between servers is a powerful technique for efficient file transfers, especially when local storage is limited. At Transloadit, our[🤖 /file/compress](/docs/robots/file-compress.md) Robot provides robust server-side file archiving into formats like `tar` (optionally gzipped) and `zip`, which can be part of a larger workflow for managing and transferring files efficiently.
[#devtips](/devtips.md)#tar-streaming#ssh-tunnel#remote-server-transfer#server-migration#backup-streaming#file-compressing-service
### 👩💻 Join 20k+ developers
Sign up for our [monthly newsletter](/newsletters.md) to receive direct links to 3 exclusive tech — and 2 product updates. No less, no more.
Your email:
Get access
## File uploading and encoding. Made simple.
Transloadit streamlines file handling for developers, trusted by brands like Coursera and The New York Times. We’re known for a reliable API, top-notch support, and a strong commitment to open source, with projects like [Uppy](https://uppy.io) and [Tus](https://tus.io) setting standards in file processing.
[Sign up](/c/)[Book a Demo](https://survey.typeform.com/to/kRg47Xi5)
No credit card needed · 5 GB included in the free plan
Cancel anytime