Import files from Backblaze in Go: efficient techniques
This example imports a prefix from an existing Backblaze B2 bucket into a new private local directory. It uses Blazer’s paginated Native API client and verifies each file’s size and whole-file SHA-1 before publishing it locally. Numbered filenames keep remote keys out of filesystem paths.
Understanding the Backblaze B2 API
The original kurin/blazer repository is no longer maintained. Its successor is
Backblaze’s Blazer repository. This example pins its current
tagged release, v0.8.0, which uses Native API v3. It is a Native API integration for an existing
bucket; Blazer’s maintainers recommend an S3 client for integrations that do not need this API.
Use an existing v3-compatible application key. Multi-Bucket Application Keys require v4 authorization and do not work with this release, even if they select just one bucket. Backblaze’s API version policy describes that distinction and continued support for older API versions.
Setting up your Go environment for Backblaze file importing
The commands below use Bash on Linux and were tested with Go 1.27.3. Install the latest security patch in the Go 1.27 line from Go’s downloads. The B2 package has no external runtime dependencies.
Paste this from a writable directory. The subshell preserves your current directory and stops on
failure; an existing backblaze-import directory is left alone. GOWORK=off keeps an enclosing Go
workspace out of this standalone project.
(
command -v go >/dev/null &&
mkdir backblaze-import &&
cd backblaze-import &&
GOWORK=off GOTOOLCHAIN=local go mod init backblaze-import &&
GOWORK=off GOTOOLCHAIN=local go get github.com/Backblaze/blazer/b2@v0.8.0
)
Using the Go SDK to connect to Backblaze B2
Set B2_KEY_ID to your application key ID and B2_APPLICATION_KEY to its secret using your secret
manager or shell environment. The key needs access to the existing bucket and the listBuckets,
listFiles, and readFiles capabilities. Bucket lookup passes the bucket’s name, so it also works
with a single-bucket key. If the key restricts filenames, B2_PREFIX must stay within that
restriction. The first credential argument is the key ID, not the bucket ID. See the
bucket lookup requirements.
Set B2_BUCKET to the bucket’s name, B2_PREFIX to a nonempty prefix, and LOCAL_PARENT to an
existing writable local directory. Include the trailing / when selecting a virtual folder:
invoices/ excludes invoices-old/. The bucket must already contain the objects you want to import,
and those objects must have a whole-file SHA-1 in their metadata. Keep this prefix stable while the
command runs.
Implementing basic file import functionality in Go
Save this complete program as backblaze-import/main.go. Each download is checked against the size
and SHA-1 from its metadata, synced, closed and renamed within the same private directory. SHA-1 here checks transfer
integrity; it is not a signature proving who created the file.
package main
import (
"context"
"crypto/sha1"
"encoding/hex"
"errors"
"fmt"
"io"
"os"
"os/signal"
"path/filepath"
"strings"
"time"
"github.com/Backblaze/blazer/b2"
)
func importObject(ctx context.Context, object *b2.Object, directory string, number int) error {
attrs, err := object.Attrs(ctx)
if err != nil {
return err
}
digest, err := hex.DecodeString(attrs.SHA1)
if err != nil || len(digest) != sha1.Size || attrs.Size < 0 {
return errors.New("object needs a whole-file SHA-1; large files without one are unsupported")
}
reader := object.NewReader(ctx)
reader.ConcurrentDownloads = 1
reader.ChunkSize = 1 << 20
defer reader.Close()
tmp, err := os.CreateTemp(directory, ".import-")
if err != nil {
return err
}
defer os.Remove(tmp.Name())
defer tmp.Close()
hash := sha1.New()
n, err := io.Copy(io.MultiWriter(tmp, hash), io.LimitReader(reader, attrs.Size+1))
if err != nil {
return err
}
if n != attrs.Size || hex.EncodeToString(hash.Sum(nil)) != strings.ToLower(attrs.SHA1) {
return errors.New("object changed or download integrity check failed")
}
if err := reader.Close(); err != nil {
return err
}
if err := tmp.Sync(); err != nil {
return err
}
if err := tmp.Close(); err != nil {
return err
}
if err := ctx.Err(); err != nil {
return err
}
// Remote names never become local paths; even ../ and absolute keys stay contained.
name := fmt.Sprintf("%06d.bin", number)
if err := os.Rename(tmp.Name(), filepath.Join(directory, name)); err != nil {
return err
}
fmt.Printf("%s\t%q\t%d bytes\n", name, object.Name(), n)
return nil
}
func importPrefix(ctx context.Context, client *b2.Client, bucketName, prefix, parent string) (string, error) {
bucket, err := client.Bucket(ctx, bucketName)
if err != nil {
return "", err
}
if bucket == nil {
return "", errors.New("bucket lookup returned no bucket")
}
directory, err := os.MkdirTemp(parent, "b2-import-")
if err != nil {
return "", err
}
// Keep successful files after a later error. Never erase an earlier import.
iterator := bucket.List(ctx, b2.ListPrefix(prefix), b2.ListPageSize(2))
number := 0
for iterator.Next() {
if err := ctx.Err(); err != nil {
return directory, err
}
number++
if err := importObject(ctx, iterator.Object(), directory, number); err != nil {
return directory, fmt.Errorf("object %d: %w", number, err)
}
}
if err := iterator.Err(); err != nil {
return directory, err
}
if err := ctx.Err(); err != nil {
return directory, err
}
fmt.Printf("Imported %d verified files\n", number)
return directory, nil
}
func run() error {
if len(os.Args) != 4 {
return errors.New("usage: backblaze-import BUCKET PREFIX EXISTING_LOCAL_PARENT")
}
if os.Args[1] == "" || os.Args[2] == "" {
return errors.New("bucket and prefix must be nonempty")
}
keyID, key := os.Getenv("B2_KEY_ID"), os.Getenv("B2_APPLICATION_KEY")
if keyID == "" || key == "" {
return errors.New("set B2_KEY_ID and B2_APPLICATION_KEY")
}
timeout := 30 * time.Minute
if value := os.Getenv("B2_IMPORT_TIMEOUT"); value != "" {
var err error
timeout, err = time.ParseDuration(value)
if err != nil || timeout <= 0 {
return errors.New("B2_IMPORT_TIMEOUT must be a positive duration, such as 30m")
}
}
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt)
defer stop()
ctx, cancel := context.WithTimeout(ctx, timeout)
defer cancel()
client, err := b2.NewClient(ctx, keyID, key)
if err != nil {
return err
}
directory, err := importPrefix(ctx, client, os.Args[1], os.Args[2], os.Args[3])
if directory != "" {
fmt.Fprintln(os.Stderr, "Import directory:", directory)
}
return err
}
func main() {
if err := run(); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
}
(
cd backblaze-import &&
GOWORK=off GOTOOLCHAIN=local go build -o backblaze-import . &&
./backblaze-import "${B2_BUCKET:?Set B2_BUCKET}" "${B2_PREFIX:?Set B2_PREFIX}" "${LOCAL_PARENT:?Set LOCAL_PARENT}"
)
The command prints the import directory on standard error. Standard output maps each numbered file
to its quoted B2 key and byte count, followed by Imported N verified files on success. Exit status
zero means every object returned by the listing passed verification. An empty matching prefix is a
successful import of zero files. Keep the mapping with the files if you need the original names.
Handling directories and batch imports: best practices
B2 object names are keys, not trusted filesystem paths. This example writes numbered .bin files and
prints a mapping from each local filename to its quoted remote key. An object named ../report or
/absolute/path therefore cannot escape the fresh import directory. The iterator fetches subsequent
pages automatically; iterator.Err() must be checked after iteration. The example requests two
names per page so a small collection exercises pagination. This costs more listing requests than a
larger page; it does not limit the number of imported objects.
Error handling and debugging: ensuring smooth file transfers
Blazer handles its own HTTP retry classification and token refresh. Do not layer an unconditional
retry loop around it: permission failures and missing buckets must be returned. All SDK calls share
a cancellable context with a thirty-minute overall deadline, including authentication, streamed
bodies, and the SDK’s backoff waits. A slow transfer can reach that deadline regardless of file size. A
failed object’s temporary file is removed; earlier successful objects remain available.
Set B2_IMPORT_TIMEOUT to a positive Go duration, such as 2h, before running the command when the
whole import needs a different deadline. Building the executable happens before this timer starts.
Progress tracking for large file transfers
The program prints one completion record per verified file. It does not claim that bytes received so far are a completed import. For objects uploaded through B2’s large-file interface, a whole-file SHA-1 may be absent; this example fails explicitly instead of reporting unchecked data as verified.
Common pitfalls and troubleshooting tips
Keep the source prefix stable during the import. A listing is not an atomic snapshot of a changing bucket; size and checksum checks catch changed content during each download but do not freeze the listing itself.
Memory management
The SDK reader uses one concurrent download and 1 MiB chunks. Files stream to disk rather than accumulating in a byte slice. Only one object is processed at a time, and each reader is closed on both success and failure.
Rate limiting
Sequential object processing bounds request concurrency. The pinned client honors retry backoff for retryable responses; the overall context deadline bounds the total wait. If the account is throttled persistently, reduce workload or schedule another run after resolving the limit.
Handling network interruptions
Ctrl+C cancels the actual SDK requests and retry waits. Cancellation is not implemented by abandoning a goroutine while a download continues writing. A later run starts a new import directory rather than overwriting partial results from an earlier batch.
Keep object downloads sequential
Keep this batch loop sequential unless measurements justify parallel object imports. The SDK already
exposes bounded chunk concurrency through Reader.ConcurrentDownloads. Increasing it also increases
memory consumption and request pressure; it does not implement an account-wide request-rate limit.
Leave bucket provisioning separate
Resolve an existing bucket before creating local output. Missing buckets and permission failures
return errors, and a nil bucket is rejected defensively. Bucket creation and lifecycle changes
belong in a separate provisioning step, so a typo during an import cannot create an unintended
resource.
Choose the next step
Use this command when your Go application needs verified local copies from B2’s Native API. If the next step is a hosted processing workflow, Transloadit’s file importing service describes the available import options.