Ensuring the integrity of downloaded files is crucial for security and reliability. In this DevTip, we'll explore how to securely verify remote file integrity using two powerful command-line tools: cURL and b2sum. We'll cover practical examples, automation techniques, and real-world use cases.

Why file integrity matters

File integrity verification ensures that files haven't been altered or corrupted during transfer. This is especially important when downloading software, updates, or sensitive data. Verifying file integrity helps protect against malicious tampering and accidental corruption.

Introducing cURL and b2sum

  • cURL: A versatile command-line tool for transferring data using various protocols, including HTTP, HTTPS, FTP, and more.
  • b2sum: A hashing utility implementing the BLAKE2 algorithm, known for its speed and security advantages over traditional hashing algorithms like MD5 or SHA-1.

Version compatibility

  • b2sum: Available in GNU Coreutils 8.24 and later.
  • cURL: All recent versions support the options used.
  • For older systems without b2sum, consider using sha256sum (widely available) or installing newer Coreutils.

Download files with cURL

To download a file securely, use the following command:

curl -fsSLo example.tar.gz https://example.com/example.tar.gz
  • -f: Fail silently on HTTP errors (important for scripts to handle errors properly).
  • -s: Silent mode - no progress meter.
  • -S: Show error messages (important when used with -s).
  • -L: Follow redirects (be cautious with untrusted sources).
  • -o: Output to file (specify exact filename for security).

Note: When downloading files from untrusted sources, always verify the hash before executing or using the downloaded content.

Generate and verify file hashes with b2sum

After downloading, generate a hash with b2sum:

b2sum example.tar.gz

This outputs a hash like:

130e76babb02d4fdfa14b33dfd2bde3914df6075eaa24fe994d776b4ced30b424e52e6fc26aad620e88ea78232bdfed339669651e83a2eca3c56cc50c5c2559c  example.tar.gz

To verify a file against a known hash, save the expected hash to a file (hash.txt):

130e76babb02d4fdfa14b33dfd2bde3914df6075eaa24fe994d776b4ced30b424e52e6fc26aad620e88ea78232bdfed339669651e83a2eca3c56cc50c5c2559c  example.tar.gz

Then verify:

b2sum -c hash.txt

If the file is intact, you'll see:

example.tar.gz: OK

Automate integrity checks in scripts

Automating file integrity checks is straightforward. Here's a robust Bash script example with proper error handling:

#!/bin/bash
set -euo pipefail  # Fail on errors, undefined vars, and pipeline failures

URL="https://example.com/example.tar.gz"
EXPECTED_HASH="130e76babb02d4fdfa14b33dfd2bde3914df6075eaa24fe994d776b4ced30b424e52e6fc26aad620e88ea78232bdfed339669651e83a2eca3c56cc50c5c2559c"
FILE="example.tar.gz"

# Create temporary file
TMP_FILE="$(mktemp)"
trap 'rm -f "$TMP_FILE"' EXIT

# Download with proper error handling
if ! curl -fsSLo "$TMP_FILE" "$URL"; then
    echo "Download failed!" >&2
    exit 1
fi

# Verify hash before moving to final location
if echo "$EXPECTED_HASH  $TMP_FILE" | b2sum -c -; then
    mv "$TMP_FILE" "$FILE"
    echo "File integrity verified."
else
    echo "File integrity check failed!" >&2
    exit 1
fi

This script downloads the file to a temporary location, verifies its integrity, and only moves it to the final location if verification succeeds.

Security considerations

  1. Always download from trusted sources over HTTPS.
  2. Verify hash checksums before using downloaded files.
  3. Be cautious with redirect following (-L option).
  4. Use specific versions in production scripts.
  5. Consider using GPG signatures for additional security.

Practical examples and use cases

  • Software Distribution: Ensure software packages haven't been tampered with.
  • Automated Deployments: Verify files before deployment in CI/CD pipelines.
  • Backup Verification: Confirm backups are intact and unaltered.

B2sum at Transloadit

At Transloadit, our 🤖 /file/hash Robot supports multiple hashing algorithms, including BLAKE2 (b2). While SHA-256 is the default algorithm, you can specify 'b2' as the algorithm parameter to use BLAKE2 hashing in your Assembly Instructions when using our Media Cataloging service.

By integrating these simple yet powerful tools into your workflow, you can significantly enhance the security and reliability of your file handling processes.