Last updated: February 5, 2025

<span aria-hidden="true" id="speed-up-tar-archiving-with-zstd-and-lz4-compression"></span>

# Speed up 'Tar' archiving with zstd and LZ4 compression

![Kevin van Zonneveld](/assets/images/teammates/avatar-kvz-4.jpg?dpl=dpl_3fBRD5jmFtSDABLXGJJyXU1nVXf8)

**Kevin van Zonneveld**

Co-founder · Amsterdam, The Netherlands · Show bio

[](https://x.com/kvz)[](https://github.com/kvz)

File archiving is essential in development workflows for backups, deployments, and data transfer. The 'tar' command, a staple tool in Unix-like systems, is commonly used for file archiving. While traditional compression methods like gzip or bzip2 can be slow with large files, modern compression algorithms offer significant performance improvements. This guide explores advanced 'tar' techniques focusing on speed optimization with Zstd and LZ4.

<span aria-hidden="true" id="installing-compression-tools"></span>

### Installing compression tools

Before using Zstd and LZ4, install them on your system:

```bash
# Ubuntu/Debian
sudo apt-get install zstd lz4

# Centos/rhel
sudo yum install zstd lz4

# macOS
brew install zstd lz4

```

<span aria-hidden="true" id="basic-tar-commands-for-file-archiving-and-extraction"></span>

### Basic 'Tar' commands for file archiving and extraction

Review the fundamental 'tar' commands for archiving and extraction.

<span aria-hidden="true" id="creating-an-archive"></span>

#### Creating an archive

```bash
tar -cf archive.tar /path/to/directory

```

Explanation:

* `-c`: Create a new archive
* `-f`: Specify the filename of the archive

<span aria-hidden="true" id="extracting-an-archive"></span>

#### Extracting an archive

```bash
tar -xf archive.tar

```

Explanation:

* `-x`: Extract files from an archive
* `-f`: Specify the filename of the archive

<span aria-hidden="true" id="compression-algorithm-comparison"></span>

### Compression algorithm comparison

Different compression methods offer various trade-offs:

* Zstd: Offers the best balance between speed and compression ratio
* LZ4: Provides the fastest compression and decompression, albeit with a lower compression ratio
* Gzip: Delivers good compression but is slower than Zstd or LZ4
* Bzip2: Achieves high compression but is significantly slower

<span aria-hidden="true" id="using-modern-compression-with-tar"></span>

### Using modern compression with 'Tar'

Modern 'tar' versions support Zstd and LZ4 directly, but note that not all versions include this feature. If your version of 'tar' does not support `--zstd` or `--lz4`, use the piped command examples below.

<span aria-hidden="true" id="using-zstd-compression"></span>

#### Using zstd compression

For newer tar versions with direct Zstd support:

```bash
tar --zstd -cf archive.tar.zst /path/to/directory

```

For older tar versions:

```bash
tar -cf - /path/to/directory | zstd > archive.tar.zst

```

To extract:

```bash
# For newer Tar versions
tar --zstd -xf archive.tar.zst

# For older Tar versions
zstd -dc archive.tar.zst | tar -xf -

```

<span aria-hidden="true" id="using-lz4-compression"></span>

#### Using LZ4 compression

For newer tar versions with direct LZ4 support:

```bash
tar --lz4 -cf archive.tar.lz4 /path/to/directory

```

For older tar versions:

```bash
tar -cf - /path/to/directory | lz4 > archive.tar.lz4

```

To extract:

```bash
# For newer Tar versions
tar --lz4 -xf archive.tar.lz4

# For older Tar versions
lz4 -dc archive.tar.lz4 | tar -xf -

```

<span aria-hidden="true" id="excluding-files-and-directories"></span>

### Excluding files and directories

Exclude specific files or patterns during archiving:

```bash
tar -cf archive.tar /path/to/directory \
  --exclude='*.log' \
  --exclude='tmp/*' \
  --exclude='node_modules'

```

<span aria-hidden="true" id="incremental-backups"></span>

### Incremental backups

Perform incremental backups using snapshot files. The snapshot file (e.g., `backup.snar`) tracks changes between backups, where a Level 0 backup is a full backup and subsequent levels store incremental changes.

```bash
# Level 0 (full) backup
tar --create \
    --file=backup-0.tar \
    --listed-incremental=backup.snar \
    /path/to/directory

# Level 1 (incremental) backup
tar --create \
    --file=backup-1.tar \
    --listed-incremental=backup.snar \
    /path/to/directory

```

<span aria-hidden="true" id="selective-archiving-with-find"></span>

### Selective archiving with 'find'

Safely archive files matching specific criteria using null-terminated inputs to handle filenames with spaces and special characters:

```bash
# Archive files modified in the last 7 days
find /path/to/directory -type f -mtime -7 -print0 | \
tar --null -cf archive.tar --files-from=-

```

<span aria-hidden="true" id="error-handling-and-verification"></span>

### Error handling and verification

Verify archive integrity and handle errors:

```bash
# Test archive integrity
tar -tf archive.tar.zst

# Extract with error checking
tar --extract \
    --file=archive.tar.zst \
    --warning=no-timestamp

```

<span aria-hidden="true" id="remote-archiving-with-ssh"></span>

### Remote archiving with ssh

Create and extract archives over SSH:

```bash
# Create remote archive
ssh user@remotehost "tar -cf - /path/to/directory" | \
zstd > archive.tar.zst

# Extract remote archive
ssh user@remotehost "zstd -dc archive.tar.zst" | \
tar -xf -

```

<span aria-hidden="true" id="splitting-large-archives"></span>

### Splitting large archives

Handle large archives by splitting them into manageable chunks:

```bash
# Create and split archive (100MB chunks)
tar -cf - /path/to/directory | \
zstd | \
split -b 100M - archive.tar.zst.part-

# Reassemble and extract
cat archive.tar.zst.part-* | \
zstd -d | \
tar -xf -

```

<span aria-hidden="true" id="automating-backups"></span>

### Automating backups

Create a backup script with error handling:

```bash
#!/bin/bash
set -euo pipefail

DATE=$(date +%F)
BACKUP_DIR="/path/to/directory"
ARCHIVE_DIR="/path/to/backup"
ARCHIVE_NAME="backup-$DATE.tar.zst"
LOG_FILE="$ARCHIVE_DIR/backup-$DATE.log"

# Create backup with logging
tar --create \
    --file=- "$BACKUP_DIR" | \
zstd > "$ARCHIVE_DIR/$ARCHIVE_NAME" 2>> "$LOG_FILE"

# Verify archive
tar -tf "$ARCHIVE_DIR/$ARCHIVE_NAME" >> "$LOG_FILE" 2>&1

```

Schedule with cron:

```cron
0 0 * * * /path/to/backup.sh

```

<span aria-hidden="true" id="conclusion"></span>

### Conclusion

Modern compression algorithms like Zstd and LZ4 significantly improve 'tar' archiving performance. Combining these tools with proper error handling, verification, and automation can help you build robust and efficient backup solutions for your development workflow.

At Transloadit, we leverage efficient file archiving and modern compression methods in our[File Compressing](/docs/robots/file-compress.md) service to handle large volumes of data effectively.

\#tar#compression#zstd#lz4#developers#file-compressing-service

### 👩‍💻 Join 20k+ developers

Sign up for our [monthly newsletter](/newsletters.md) to receive direct links to 3 exclusive tech — and 2 product updates. No less, no more.

Your email:

Get access

## File uploading and encoding. Made simple.

Transloadit streamlines file handling for developers, trusted by brands like Coursera and The New York Times. We’re known for a reliable API, top-notch support, and a strong commitment to open source, with projects like [Uppy⁠](https://uppy.io) and [Tus⁠](https://tus.io) setting standards in file processing.

[Sign up](/c/)[Book a Demo](https://survey.typeform.com/to/kRg47Xi5)

No credit card needed · 5 GB included in the free plan

Cancel anytime