Hash files in parallel in Rust with Rayon and SHA-256
To hash a batch of files in Rust, stream each file through SHA-256 and let Rayon process separate files concurrently. The command below prints a standard SHA-256 checksum for each file without loading whole files into memory. Parallel workers may improve batch throughput, but a single file still runs sequentially, and storage can limit any speedup.
System requirements
This walkthrough uses Bash on Linux, Rust and Cargo 1.98.1, ring 0.17.14, and Rayon 1.12.0. It was
tested in a Debian 12, ARM64 container. Install the Rust toolchain and a
C compiler and linker for building ring; the verification step also uses GNU sha256sum.
Cargo needs network access to download the dependencies on the first build.
Create the project
Run this from the parent directory where you want a new file-hashing project:
cargo new --bin --edition 2024 --vcs none file-hashing &&
cd file-hashing &&
cargo add ring@=0.17.14 rayon@=1.12.0
The && chain stops if project creation or navigation fails. If file-hashing already exists,
Cargo refuses to replace it. Choose an unused location and rerun the whole block. Continue with the
remaining steps only after setup succeeds, staying inside the new project directory.
The = requirements pin the two direct dependencies. Keep the generated
Cargo.lock alongside the
manifest to preserve the resolved versions of their dependencies too.
Use the checksum algorithm your recipient expects
This example uses SHA-256 so you can compare its output directly with sha256sum. A BLAKE2 digest
would not match a SHA-256 reference, regardless of which library computes it. Choose the algorithm
before optimizing its implementation.
Stream each file and parallelize the batch
Replace the generated src/main.rs with this complete program. This intentionally replaces Cargo’s
placeholder source:
use rayon::prelude::*;
use ring::digest::{Context, SHA256};
use std::fs::File;
use std::io::{self, ErrorKind, Read};
use std::path::PathBuf;
use std::process::ExitCode;
fn sha256_digest<R: Read>(mut reader: R) -> io::Result<String> {
let mut context = Context::new(&SHA256);
let mut buffer = [0u8; 8192];
loop {
let count = match reader.read(&mut buffer) {
Ok(count) => count,
Err(error) if error.kind() == ErrorKind::Interrupted => continue,
Err(error) => return Err(error),
};
if count == 0 {
break;
}
context.update(&buffer[..count]);
}
Ok(context
.finish()
.as_ref()
.iter()
.map(|byte| format!("{byte:02x}"))
.collect())
}
fn main() -> ExitCode {
let files: Vec<PathBuf> = std::env::args_os().skip(1).map(PathBuf::from).collect();
if files.is_empty() {
eprintln!("Usage: file-hashing <file> [file ...]");
return ExitCode::from(2);
}
let result = files.par_iter().try_for_each(|path| -> io::Result<()> {
let digest = File::open(path).and_then(sha256_digest).map_err(|error| {
io::Error::new(error.kind(), format!("{}: {error}", path.display()))
})?;
println!("{digest} {}", path.display());
Ok(())
});
match result {
Ok(()) => ExitCode::SUCCESS,
Err(error) => {
eprintln!("{error}");
ExitCode::FAILURE
}
}
}
Each worker keeps an 8 KiB read buffer and its own
ring::digest::Context.
Repeated update calls hash the file’s bytes in order; finish produces the complete digest.
The loop hashes only the bytes actually read, continues after short reads, and retries
ErrorKind::Interrupted.
Other read errors return a failure instead of a checksum for incomplete data.
Rayon’s
try_for_each
schedules different paths concurrently. Output order is unspecified. If opening or reading a file
fails, the command reports the path on stderr and exits with status 1. Rayon tries to stop the
remaining work, but other workers may already have printed checksums; treat the batch as incomplete.
With no arguments, the command prints usage and exits with status 2.
Run it and check the digests
Create two small fixtures and run the program. The subshell’s set -C refuses to overwrite existing
sample files, and each && prevents later commands from running after a failure:
(
set -C
printf abc > abc.txt &&
: > empty.txt &&
cargo run --release --locked -- abc.txt empty.txt
)
Expect these two lines, in either order. The abc.txt fixture contains three bytes, with no newline:
ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad abc.txt
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 empty.txt
Compare the digests with an independent implementation:
sha256sum abc.txt empty.txt
To repeat the hash operation, reuse the files:
cargo run --release --locked -- abc.txt empty.txt
The command only reads its inputs and prints to stdout. Cargo may rebuild the executable in
target/release/; it does not replace your input files. Pass your own file paths after --, quoting
paths that contain spaces. The printed paths are for display: this example does not implement
sha256sum’s filename escaping or checksum verification mode.
Compare worker counts on your storage
Once the release build exists, time the executable directly so compilation is excluded. Rayon 1.12
uses RAYON_NUM_THREADS
to set its worker count:
time env RAYON_NUM_THREADS=1 ./target/release/file-hashing abc.txt empty.txt &&
time env RAYON_NUM_THREADS=4 ./target/release/file-hashing abc.txt empty.txt
These tiny fixtures check the commands, not performance. Replace both commands’ paths with the same representative batch before comparing elapsed times. Repeat the runs and alternate their order: the operating system’s file cache can make a later run faster. More workers can help when hashing is the bottleneck, or slow things down when concurrent reads compete for storage bandwidth. Use the worker count that helps your workload; there is no promised speedup.
Compare against a trusted reference
To verify a download, compare the digest with the publisher’s expected SHA-256 from a trusted source. A checksum alone does not authenticate a file if someone can replace both the file and its reference. For example, Ubuntu verifies its checksum file’s signature before checking the download.
Keep files unchanged while hashing them: this program reads a stream, so it cannot give you a consistent snapshot of a file that another process is modifying.