Accelerating file hashing in Rust with parallel processing

Kevin van Zonneveld

Co-founder · Amsterdam, The Netherlands · Show bio

Hashing files is a fundamental task in software development, crucial for data integrity, security, and efficient data management. In Rust, you can leverage open-source libraries like ring and RustCrypto to implement robust and efficient file hashing. In this DevTip, we explore how to hash files in Rust using these libraries, compare different hashing algorithms such as SHA-256 and BLAKE2, and provide practical code examples to get you started.

Introduction to file hashing in Rust

File hashing generates a fixed-size digest from file data. Different inputs can share a digest, although collisions should be infeasible to find with a secure cryptographic hash. Hashing verifies file integrity, detects duplicates, supports cryptographic operations, and more. Rust, with its performance and safety guarantees, is an excellent choice for implementing file hashing in your applications.

System requirements

• Current stable Rust (current Rayon 1.x releases require Rust 1.80 or later)
• A C compiler (gcc, clang, or MSVC on Windows)
• pkg-config (on Unix-like systems)

Setting up your Rust environment

First, ensure you have the latest stable version of Rust installed. You can download it from the official website⁠ or update your existing installation with:

rustup update stable

Create a new Rust project:

cargo new file-hashing
cd file-hashing

Choosing the right hashing algorithm

Selecting the appropriate hashing algorithm depends on your application's requirements:

• SHA-256: A cryptographic hash function offering high security and widely used in various applications.
• BLAKE2: A modern, faster alternative to SHA algorithms, providing comparable security with improved performance.

Implementing file hashing with ring

The ring⁠ crate provides safe and fast cryptographic operations, including hashing.

Add ring to your Cargo.toml:

[dependencies]
ring = "0.17.8"

Hashing a file using ring

Below is an example of hashing a file using SHA-256 with ring:

use ring::digest::{Context, SHA256};
use std::io::{ErrorKind, Read};

fn sha256_digest<R: Read>(mut reader: R) -> Result<String, std::io::Error> {
    let mut context = Context::new(&SHA256);
    let mut buffer = [0u8; 8192];

    loop {
        let count = match reader.read(&mut buffer) {
            Ok(count) => count,
            Err(error) if error.kind() == ErrorKind::Interrupted => continue,
            Err(error) => return Err(error),
        };
        if count == 0 {
            break;
        }
        context.update(&buffer[..count]);
    }

    Ok(context.finish().as_ref().iter().map(|byte| format!("{byte:02x}")).collect())
}

Explanation

• Context and Digest: ring::digest::Context manages incremental hash computation, and the final hash is produced via context.finish().
• Reading the File: The file is read in chunks to efficiently handle large files.
• Error Handling: The function returns a Result, allowing you to handle I/O errors, such as file access issues.

Additional features of ring

Beyond hashing, ring offers various cryptographic functions including encryption, digital signatures, and key agreement protocols. It is designed to be secure and performant, making it suitable for applications with high security requirements.

Exploring advanced hashing with RustCrypto

For a wider range of algorithms and added flexibility, the RustCrypto project provides several hashing crates.

Add the desired hash function crate to your Cargo.toml. For example, to use BLAKE2:

[dependencies]
blake2 = "0.10.6"

Implementing Blake2 hashing

This example demonstrates how to hash a file using BLAKE2 with RustCrypto:

use blake2::{Blake2b512, Digest};
use std::io::{ErrorKind, Read};

fn blake2b_digest<R: Read>(mut reader: R) -> Result<String, std::io::Error> {
    let mut hasher = Blake2b512::new();
    let mut buffer = [0u8; 8192];

    loop {
        let count = match reader.read(&mut buffer) {
            Ok(count) => count,
            Err(error) if error.kind() == ErrorKind::Interrupted => continue,
            Err(error) => return Err(error),
        };
        if count == 0 {
            break;
        }
        hasher.update(&buffer[..count]);
    }

    let result = hasher.finalize();
    Ok(format!("{:x}", result))
}

Explanation

• Blake2b512: Implements the BLAKE2b hash function with a 512-bit output.
• Reading and Hashing: Similar to the ring example, the file is processed in chunks to efficiently compute the hash.

Comparative analysis of hashing methods

When choosing a hashing algorithm, consider the following factors:

• Security: SHA-256 and BLAKE2 provide robust security for cryptographic purposes.
• Performance: BLAKE2 often outperforms SHA-256, offering faster hashing with similar security.
• Compatibility: For interoperability, choose an algorithm that is widely supported across platforms.

Best practices for file hashing in Rust projects

• Use Buffered Reading: Employ a BufReader to optimize file I/O performance.
• Handle Errors Gracefully: Utilize Rust's error handling (using the Result type) to manage I/O and other errors.
• Avoid Blocking I/O: For applications processing multiple files, consider asynchronous I/O or parallel processing to improve throughput.

Accelerating file hashing with parallel processing

Hashing files sequentially can be inefficient when dealing with multiple files. With the rayon crate, you can process files concurrently.

Add rayon alongside ring in your existing Cargo.toml dependency section (do not duplicate the [dependencies] heading). Keep Cargo.lock to reproduce your application's dependency versions:

[dependencies]
rayon = "1.10.0"
ring = "0.17.8"

Parallel file hashing example

Put the sha256_digest function and its imports from the earlier ring example above this main function in src/main.rs. Run it with cargo run --release -- file1.bin file2.bin.

use rayon::prelude::*;
use std::fs::File;
use std::path::PathBuf;
use std::io::{self, BufReader};

fn main() -> Result<(), io::Error> {
    let files: Vec<PathBuf> = std::env::args_os()
        .skip(1)
        .map(PathBuf::from)
        .collect();

    if files.is_empty() {
        eprintln!("Usage: {} <file1> <file2> ...", env!("CARGO_PKG_NAME"));
        return Err(io::Error::new(io::ErrorKind::InvalidInput, "No input files provided"));
    }

    files.par_iter()
         .try_for_each(|file| -> Result<(), io::Error> {
             let input = File::open(file)?;
             let reader = BufReader::new(input);
             let digest = sha256_digest(reader)?;
             println!("{} {}", digest, file.display());
             Ok(())
         })
}

Performance considerations

This example hashes different files concurrently; it does not parallelize chunks of one file. Every digest still covers the complete file in byte order. A single file gains no parallel speedup here. For multiple files, benchmark on your storage device: I/O contention and scheduling overhead can outweigh any gain. Output order is unspecified, and an I/O error stops the operation with an error, potentially after other file hashes have already been printed.

How file hashing contributes to data integrity and security

• Data Integrity: Compare a file's digest against a trusted reference to detect changes during transmission or storage. An attacker who can replace both the file and its reference hash can defeat this check.
• Security: Cryptographic hashes are building blocks for authentication and digital signatures. Plain SHA-256 or BLAKE2 is not suitable for password storage; use a password hashing scheme such as Argon2id for that purpose.
• Deduplication: Hashing enables the identification of duplicate files, optimizing storage use.

Conclusion

File hashing in Rust is both straightforward and efficient with the help of open-source libraries such as ring and RustCrypto. By choosing the right hashing algorithm and leveraging parallel processing via Rayon, you can build robust applications for ensuring data integrity, security, and efficient data management. For additional file processing solutions, consider exploring Transloadit's Media Cataloging service.