File integrity verification is essential for ensuring that downloaded files have not been corrupted or tampered with during transfer. In this guide, we demonstrate a two-step process: first, use cURL to download both the file and its corresponding checksum file, and then verify the file's integrity using system utilities such as sha256sum.

Understanding file hashes

A file hash is a unique fixed-size string that serves as a digital fingerprint of file content. Common hash algorithms include:

  • SHA-256 (recommended for general use)
  • SHA-512 (for enhanced security)
  • SHA-1 (legacy—avoid for security-critical applications)
  • MD5 (legacy—avoid for secure verification)

File hashes are crucial for verifying software downloads, detecting unauthorized file modifications, ensuring data integrity, and implementing caching mechanisms.

Downloading files and their hashes

Most software distributions provide a separate checksum file alongside the download. By downloading both files, you can verify that the file you received matches the expected hash. For example:

# Download both the file and its sha-256 checksum
curl -fsSLO https://example.com/file.tar.gz
curl -fsSLO https://example.com/file.tar.gz.sha256

The -fsSLO flags ensure that cURL fails silently on HTTP errors, operates in silent mode while showing errors, follows redirects, and saves the file with the same name as in the URL.

Real-world example: verifying cURL release

Consider the example of verifying a cURL release. This example downloads the release and its checksum, then verifies the integrity using sha256sum:

# Download cURL release and its checksum
curl -fsSLO https://curl.se/download/curl-8.5.0.tar.gz
curl -fsSLO https://curl.se/download/curl-8.5.0.tar.gz.sha256

# Verify the checksum
sha256sum -c curl-8.5.0.tar.gz.sha256

This method has been verified with cURL 8.5.0 and standard Linux utilities.

Automating verification with bash

Integrate file verification into your workflow with a bash script that downloads a file and its checksum, then verifies its integrity:

#!/bin/bash

set -euo pipefail

FILE_URL="https://example.com/file.tar.gz"
FILE_NAME="$(basename "$FILE_URL")"

# Download file and checksum with retry mechanism
curl -fsSL --retry 3 -o "$FILE_NAME" "$FILE_URL"
curl -fsSL --retry 3 -o "$FILE_NAME.sha256" "$FILE_URL.sha256"

# Verify file integrity
if sha256sum -c "$FILE_NAME.sha256"; then
    echo "✓ File verification successful"
else
    echo "✗ File verification failed" >&2
    exit 1
fi

Security considerations

  • Use HTTPS: Always download files and checksums over HTTPS to protect against tampering.
  • Trust Sources: Ensure both the file and its checksum are obtained from reputable sources.
  • Hash Algorithm Selection: Prefer secure algorithms like SHA-256 or SHA-512; avoid MD5 and SHA-1.
  • Verify Authenticity: Where available, verify checksum files using GPG signatures to confirm their authenticity.

Github Actions integration

Automate file verification within your CI/CD pipeline with GitHub Actions:

name: Verify Downloaded Files

on: [push, pull_request]

jobs:
  verify:
    runs-on: ubuntu-latest
    steps:
      - name: Download File and Checksum
        run: |
          curl -OL "$DOWNLOAD_URL"
          curl -OL "$DOWNLOAD_URL.sha256"
        env:
          DOWNLOAD_URL: https://example.com/file.tar.gz

      - name: Verify Checksum
        run: sha256sum -c *.sha256

This example downloads the file and its checksum, then verifies the integrity using sha256sum.

Troubleshooting

  • Hash Mismatch Issues:
    • Verify that both files are completely downloaded (check file sizes).
    • Ensure the checksum file is properly formatted without extra whitespace or incorrect line endings.
  • Download Problems:
    • Confirm the URL is accessible and that network connectivity is stable.
    • Check that there is sufficient disk space.
  • Common Error Messages:
    • sha256sum: no properly formatted checksum lines found: Verify the checksum file format.
    • curl: (22) HTTP error: Verify the URL and necessary permissions.

Best practices

  • Always download files and checksums over HTTPS.
  • Use a secure hash algorithm like SHA-256 for verification.
  • Implement automated verification in scripts or CI/CD pipelines.
  • Monitor and update your verification scripts to address any changes in hash standards or tool deprecations.

Conclusion

Verifying file integrity is essential for secure software distribution and protecting your development workflow. By using cURL for downloading files and system utilities like sha256sum for verification, you can ensure that your data has not been tampered with. This approach can be seamlessly integrated into both manual and automated processes.

If you need robust solutions for handling file uploads and processing, Transloadit offers advanced capabilities to ensure data integrity and security throughout your workflow.

References