Verify artifact manifests with b2sum and GNU Parallel

Tim Koschützki

Co-founder · Berlin, Germany ·

Use GNU b2sum to record a completed set of artifacts, then check that set without regenerating its expected values. This walkthrough hashes a small backup directory in parallel, replaces the manifest only after every hash succeeds, and propagates verification failures to a shell or CI job.

A digest alone does not establish who created a file: obtain the expected checksum through a trusted channel, such as an authenticated release or a signed manifest, rather than downloading both file and checksum from an untrusted source.

Choose the matching checksum format

GNU b2sum defaults to BLAKE2b-512, written as 128 hexadecimal characters. Use the same algorithm and digest length as the expected values. BLAKE2s and SHA-256 are different formats; if a publisher supplies SHA-256, verify with sha256sum instead of inventing a BLAKE2 baseline from the download.

This demo creates a local baseline from known files. In a release workflow, generate the manifest on the trusted producer side after the artifacts are complete, then protect its origin and contents. Someone who can replace both artifacts and expected values can make verification pass.

Check the Linux prerequisites

Use Linux with Bash, GNU coreutils, GNU findutils, GNU Parallel, and Python 3. On Debian or Ubuntu, the corresponding packages are bash, coreutils, findutils, parallel, and python3. GNU Parallel is a separate installation; another command named parallel is not a substitute. Check availability before creating the demo directory:

bash --version && b2sum --version && find --version && parallel --version && python3 --version

The example was tested with Bash 5.3.15, coreutils 9.11, findutils 4.11.0, GNU Parallel 20240222, and Python 3.14.7. Python only creates the demo files and needs no third-party packages. This is a local Linux workflow, not a macOS or Windows recipe.

Create a small artifact set

Start in a directory you control. This command enters a new directory and refuses to reuse one that already exists:

mkdir -- integrity-demo && cd -- integrity-demo

Keep the remaining files and commands in integrity-demo. Save the following as make-fixture.py:

from pathlib import Path

root = Path('backups')
root.mkdir()
(root / 'archive').mkdir()
samples = {
    'app.tar': b'first release\n',
    'archive/app.tar': b'other release\n',
    'empty.tar': b'',
    '-draft.tar': bytes([0, 255, 16, 10]),
    'café copy.tar': b'Unicode name\n',
    'line\nbreak.tar': b'newline name\n',
}
for name, content in samples.items():
    (root / name).write_bytes(content)

Run it once:

python3 -I make-fixture.py

These six small files exercise empty and binary contents, duplicate basenames, spaces, Unicode, and a newline in a filename. They are sample bytes with a .tar suffix, not actual tar archives. The generator refuses an existing backups directory. If setup is interrupted, inspect the partial directory and start a new demo in an unused directory; rerunning the generator will not repair or overwrite the earlier files.

Generate a complete manifest

Save this as hash-backups.sh. Run it as a saved Bash script rather than pasting it into your shell:

#!/usr/bin/env bash
set -euo pipefail

export LC_ALL=C
umask 077

if (( $# != 1 )) || [[ -z $1 ]]; then
  printf 'Usage: bash hash-backups.sh BACKUP_DIR\n' >&2
  exit 2
fi
backupDir=$1
if [[ $backupDir != /* ]]; then
  backupDir=./$backupDir
fi
if [[ ! -d $backupDir ]]; then
  printf 'Not a directory: %s\n' "$backupDir" >&2
  exit 2
fi

tempDir=$(mktemp -d .checksums.XXXXXX)
trap 'rm -rf -- "$tempDir"' EXIT
find "$backupDir" -type f -name '*.tar' -print0 > "$tempDir/files.nul"
if [[ ! -s $tempDir/files.nul ]]; then
  printf 'No .tar files selected in %s\n' "$backupDir" >&2
  exit 1
fi

sort -z "$tempDir/files.nul" |
  parallel --plain --will-cite --tmpdir "$tempDir" -0 --jobs 4 \
    --keep-order --halt now,fail=1 b2sum -- {} > "$tempDir/checksums.b2"
test -s "$tempDir/checksums.b2"
mv -fT -- "$tempDir/checksums.b2" checksums.b2

Run the producer:

bash hash-backups.sh backups

It recursively selects regular files with a lowercase .tar suffix, without following symlink entries. GNU Parallel runs up to four hashing jobs, groups their output, and retains the sorted input order. Its --plain option ignores personal profiles and PARALLEL settings, while --halt now,fail=1 stops on a failed hash. An empty selection, a failed directory scan, or a failed hash exits nonzero before replacing checksums.b2.

The temporary directory is beside the destination, so the final mv publishes the completed manifest by renaming it on the same filesystem. A successful rerun replaces the previous manifest; failures leave it intact and remove temporary files. Run one producer at a time, and keep the artifact directory unchanged while scanning, hashing, or verifying. This is not a filesystem snapshot.

Verify the recorded files

Run the consumer from the same integrity-demo directory, leaving checksums.b2 unchanged:

b2sum --check --strict --quiet checksums.b2 && printf 'All recorded files match checksums.b2\n'

All recorded files match checksums.b2

--check reads the paths inside the manifest. Those paths include directories, so the two app.tar files remain distinct. Relative paths are resolved from your current directory, not the manifest’s directory. Copy the complete directory layout with the manifest when checking a transferred set.

find -print0 and parallel -0 preserve filename boundaries. The manifest itself uses GNU’s escaped newline-delimited format, which can represent newlines and backslashes in filenames. Do not split its records on whitespace or add b2sum --zero: NUL-delimited checksum output is not supported by --check.

Now change only the intended artifact and repeat the verification step:

printf 'changed release\n' >> backups/app.tar &&
  b2sum --check --strict --quiet checksums.b2

The command exits nonzero and reports a checksum mismatch for ./backups/app.tar, even though backups/archive/app.tar still exists. A missing or unreadable recorded file also fails. --strict additionally makes malformed checksum records fail. Fix the artifact or restore it from a trusted copy before checking again; regenerating the manifest would accept the changed bytes.

Verification checks the recorded members. It does not detect newly added files, check archive structure, or establish who created the artifacts. Use a new reviewed producer run when you intend to change the baseline, rather than treating extra files as already verified.

Propagate failures in CI

Save the verification command as verify-backups.sh so its status reaches the caller:

#!/usr/bin/env bash
set -euo pipefail
b2sum --check --strict checksums.b2

Run it from the directory containing the trusted manifest and its recorded layout:

bash verify-backups.sh

A CI job can use that same invocation after checking out a protected manifest and making the artifacts available at the recorded paths. Keep manifest generation out of the verification job. Review legitimate baseline updates separately, and protect write access to the manifest as well as the artifacts. Piping the check into another command without preserving its exit status can hide a mismatch.

Measure performance on your workload

Performance depends on file size, storage, caching, CPU instructions, and implementation. These commands show how to time the demo file. Replace backups/app.tar with a finished representative file for meaningful measurements:

time b2sum -- backups/app.tar > /dev/null &&
  time sha256sum -- backups/app.tar > /dev/null

The tiny demo files cannot establish a useful speed comparison. Do not assume BLAKE2 is always faster than hardware-accelerated SHA-256. Parallel hashing can contend for the same storage device, so measure worker counts on your own dataset before raising --jobs.

To verify a single downloaded file after moving it, use the separate b2sum verification walkthrough.