Build a resumable file downloader in Go with concurrent chunks
Build one executable downloader that saves completed chunks between runs and checks the assembled file before replacing the destination. You will also run a local HTTP range origin with known binary bytes to try the complete download without relying on a third-party URL.
Why build a custom file downloader?
Four workers fetch independent chunks. Completed chunks survive an interrupted run, while partial chunks never count as complete. The destination is replaced atomically only after the complete file matches the expected digest.
Understanding HTTP range requests and partial content
A Range header is a request, not a guarantee. Every chunk must receive status 206, exactly the requested Content-Range, the same strong ETag, and exactly the expected number of bytes. If-Range prevents silently combining different object versions; a fallback 200 is an error here.
HTTP range and validator semantics (RFC 9110)
Basic file download implementation in Go
These commands target Linux with Bash, Go 1.26.8 on your PATH, and sha256sum from GNU Coreutils.
Install Go from the official downloads if needed. Go 1.26 remains a
supported release; this walkthrough was tested with
1.26.8 and makes no claim about older toolchains. Both programs use only the standard library.
Paste this block from a directory you control. It enters the new project only after initialization
succeeds. An existing range-downloader directory is an error; choose another name or inspect it
instead of removing an existing project to repeat setup.
(
mkdir -- range-downloader &&
cd -- range-downloader &&
GOENV=off GOWORK=off GOTOOLCHAIN=local GOFLAGS= go mod init range-downloader
) && cd -- range-downloader
The command-scoped settings bypass saved Go configuration and an enclosing
Go workspace, clear inherited build flags, and use the installed
toolchain. They do not change your shell settings or the parent project's go.mod or go.work.
If setup fails after creating the directory, your shell stays in its original directory; inspect
the new directory before trying again.
Adding support for resumable downloads
The output path plus “.parts” stores the URL, ETag, expected digest, length and chunk size. A rerun accepts that state only if all fields match. It reuses complete chunk files of the expected length and verifies the assembled content against the trusted digest. A mismatch requires a new output path; do not reuse corrupt state.
Implementing concurrent chunk downloading
A fixed worker pool consumes an unbuffered job channel. Each response streams through a 32 KiB copy buffer into a temporary chunk file. The first worker error cancels the other requests, and the coordinator waits for every worker before returning.
Adding a progress bar with real-time updates
This version prints byte counts during final assembly instead of adding a progress-bar dependency. Downloaded bytes are not reported as verified until the final digest matches.
Error handling and retry logic
Errors propagate to a nonzero process exit. Rerun the same command after a transient failure to reuse completed chunks. There is no automatic retry loop that could keep retrying an authorization failure or changed representation. Ctrl-C cancels network requests and leaves completed chunks available.
Optimizing performance with connection pooling
One shared HTTP client reuses connections, with four idle connections per host and a two-minute timeout per request. Worker count bounds network concurrency; disk use includes the saved chunks and a second full copy during assembly.
Complete example: building a CLI downloader
Save this complete program as main.go. Obtain the SHA-256 digest from a trusted publisher
independently of the download. Use a parent directory you control; no other process should modify
the output or saved chunks.
package main
import (
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"os"
"os/signal"
"path/filepath"
"strings"
"sync"
"time"
)
const chunkSize int64 = 4 << 20
type identity struct {
URL, ETag, SHA256 string
Size, ChunkSize int64
}
func probe(ctx context.Context, client *http.Client, url, digest string) (identity, error) {
req, err := http.NewRequestWithContext(ctx, http.MethodHead, url, nil)
if err != nil {
return identity{}, err
}
req.Header.Set("Accept-Encoding", "identity")
resp, err := client.Do(req)
if err != nil {
return identity{}, err
}
defer resp.Body.Close()
tag := resp.Header.Get("ETag")
if resp.StatusCode != 200 || resp.ContentLength < 0 || resp.ContentLength > 1<<40 ||
len(tag) < 2 || !strings.HasPrefix(tag, "\"") || !strings.HasSuffix(tag, "\"") ||
resp.Header.Get("Content-Encoding") != "" {
return identity{}, errors.New("need a known size (at most 1 TiB), strong ETag and unencoded HEAD 200")
}
return identity{url, tag, digest, resp.ContentLength, chunkSize}, nil
}
func fetchChunk(ctx context.Context, client *http.Client, id identity, dir string, start int64) error {
end := min(start+id.ChunkSize, id.Size) - 1
name := filepath.Join(dir, fmt.Sprintf("%d.part", start))
if info, err := os.Lstat(name); err == nil {
if info.Mode().IsRegular() && info.Size() == end-start+1 {
return nil
}
return errors.New("invalid saved chunk; use a new output path")
} else if !errors.Is(err, os.ErrNotExist) {
return err
}
req, err := http.NewRequestWithContext(ctx, http.MethodGet, id.URL, nil)
if err != nil {
return err
}
req.Header.Set("Accept-Encoding", "identity")
req.Header.Set("Range", fmt.Sprintf("bytes=%d-%d", start, end))
req.Header.Set("If-Range", id.ETag)
resp, err := client.Do(req)
if err != nil {
return err
}
defer resp.Body.Close()
expected := fmt.Sprintf("bytes %d-%d/%d", start, end, id.Size)
if resp.StatusCode != http.StatusPartialContent || resp.Header.Get("Content-Range") != expected ||
resp.Header.Get("ETag") != id.ETag || resp.Header.Get("Content-Encoding") != "" ||
(resp.ContentLength != -1 && resp.ContentLength != end-start+1) {
return errors.New("server rejected range or changed representation")
}
tmp, err := os.CreateTemp(dir, ".chunk-")
if err != nil {
return err
}
defer os.Remove(tmp.Name())
defer tmp.Close()
n, err := io.CopyBuffer(tmp, io.LimitReader(resp.Body, end-start+2), make([]byte, 32<<10))
if err != nil {
return err
}
if n != end-start+1 {
return errors.New("incorrect chunk length")
}
if err := tmp.Sync(); err != nil {
return err
}
if err := tmp.Close(); err != nil {
return err
}
if err := ctx.Err(); err != nil {
return err
}
return os.Rename(tmp.Name(), name)
}
func download(ctx context.Context, client *http.Client, url, output, digest string, workers int) error {
sum, err := hex.DecodeString(digest)
if err != nil || len(sum) != sha256.Size || workers < 1 || workers > 16 {
return errors.New("provide a SHA-256 hex digest and 1–16 workers")
}
digest = strings.ToLower(digest)
id, err := probe(ctx, client, url, digest)
if err != nil {
return err
}
dir := output + ".parts"
fresh := false
if err := os.Mkdir(dir, 0700); err == nil {
fresh = true
} else if !errors.Is(err, os.ErrExist) {
return err
}
info, err := os.Lstat(dir)
if err != nil {
return err
}
if !info.IsDir() || info.Mode().Perm()&0077 != 0 {
return errors.New("parts directory must be private")
}
lock := filepath.Join(dir, ".lock")
if err := os.Mkdir(lock, 0700); err != nil {
return errors.New("parts directory locked; another download may be active")
}
defer os.Remove(lock)
manifest := filepath.Join(dir, "identity.json")
if fresh {
data, err := json.Marshal(id)
if err != nil {
return err
}
if err := os.WriteFile(manifest, data, 0600); err != nil {
return err
}
} else {
data, err := os.ReadFile(manifest)
if err != nil {
return err
}
var saved identity
if err := json.Unmarshal(data, &saved); err != nil {
return err
}
if saved != id {
return errors.New("saved download identity changed; use a new output path")
}
}
ctx, cancel := context.WithCancel(ctx)
defer cancel()
var wg sync.WaitGroup
var once sync.Once
var firstErr error
jobs := make(chan int64)
for i := 0; i < workers; i++ {
wg.Add(1)
go func() {
defer wg.Done()
for start := range jobs {
if ctx.Err() != nil {
return
}
if err := fetchChunk(ctx, client, id, dir, start); err != nil {
once.Do(func() { firstErr = err; cancel() })
return
}
}
}()
}
send:
for start := int64(0); start < id.Size; start += id.ChunkSize {
select {
case jobs <- start:
case <-ctx.Done():
break send
}
}
close(jobs)
wg.Wait()
if firstErr != nil {
return firstErr
}
if err := ctx.Err(); err != nil {
return err
}
tmp, err := os.CreateTemp(filepath.Dir(output), ".download-")
if err != nil {
return err
}
defer os.Remove(tmp.Name())
defer tmp.Close()
hash := sha256.New()
for start := int64(0); start < id.Size; start += id.ChunkSize {
if err := ctx.Err(); err != nil {
return err
}
part, err := os.Open(filepath.Join(dir, fmt.Sprintf("%d.part", start)))
if err != nil {
return err
}
n, copyErr := io.Copy(io.MultiWriter(tmp, hash), io.LimitReader(part, id.ChunkSize+1))
closeErr := part.Close()
if copyErr != nil {
return copyErr
}
if closeErr != nil {
return closeErr
}
if n != min(id.ChunkSize, id.Size-start) {
return errors.New("saved chunk length changed")
}
fmt.Fprintf(os.Stderr, "Assembled: %d/%d bytes\n", start+n, id.Size)
}
if hex.EncodeToString(hash.Sum(nil)) != digest {
return errors.New("SHA-256 mismatch; use a new output path")
}
if err := tmp.Sync(); err != nil {
return err
}
if err := tmp.Close(); err != nil {
return err
}
if err := ctx.Err(); err != nil {
return err
}
return os.Rename(tmp.Name(), output)
}
func main() {
if len(os.Args) != 4 {
fmt.Fprintln(os.Stderr, "usage: downloader URL OUTPUT SHA256")
os.Exit(2)
}
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt)
defer stop()
transport := http.DefaultTransport.(*http.Transport).Clone()
transport.MaxIdleConnsPerHost = 4
defer transport.CloseIdleConnections()
client := &http.Client{Transport: transport, Timeout: 2 * time.Minute}
if err := download(ctx, client, os.Args[1], os.Args[2], os.Args[3], 4); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
}
For your own download, set DOWNLOAD_URL, OUTPUT_PATH, and EXPECTED_SHA256 in this shell.
The origin must supply an unencoded HEAD response with a known size and a strong ETag, then honor
the chunk requests. A successful download deliberately replaces an existing output file.
Failed verification preserves that file. Choose the destination accordingly.
Paste the build-and-run block below, or first use the local origin in the next section. Building
main.go explicitly keeps the two standalone programs separate. A failed build stops the block
even when an older downloader executable exists; missing variables fail inside the subshell.
(
GOENV=off GOWORK=off GOTOOLCHAIN=local GOFLAGS= GOOS= GOARCH= go build -o downloader main.go &&
./downloader "${DOWNLOAD_URL:?Set DOWNLOAD_URL}" "${OUTPUT_PATH:?Set OUTPUT_PATH}" "${EXPECTED_SHA256:?Set EXPECTED_SHA256}"
)
Try a local range origin
Save this second program as origin.go beside main.go. It serves 8 MiB of repeated binary bytes
followed by the five bytes tail\n, for a total of 8,388,613 bytes. Each request gets its own reader;
http.ServeContent handles HEAD, Range, and If-Range
using the strong ETag supplied here. This small fixture keeps its content in memory.
package main
import (
"bytes"
"crypto/sha256"
"fmt"
"net"
"net/http"
"os"
"time"
)
func main() {
if len(os.Args) != 2 {
fmt.Fprintln(os.Stderr, "usage: local-origin 127.0.0.1:PORT")
os.Exit(2)
}
data := append(bytes.Repeat([]byte{0x00, 0x80, 0xff, 0x0a}, 2<<20), []byte("tail\n")...)
digest := fmt.Sprintf("%x", sha256.Sum256(data))
listener, err := net.Listen("tcp", os.Args[1])
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
defer listener.Close()
mux := http.NewServeMux()
mux.HandleFunc("/file", func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("ETag", "\""+digest+"\"")
http.ServeContent(w, r, "fixture.bin", time.Time{}, bytes.NewReader(data))
})
fmt.Printf("DOWNLOAD_URL=http://%s/file\nEXPECTED_SHA256=%s\n", listener.Addr(), digest)
server := &http.Server{Handler: mux, ReadHeaderTimeout: 5 * time.Second}
if err := server.Serve(listener); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
}
In a second Bash terminal, enter the same range-downloader directory and paste this block.
Port zero selects an available loopback port. Keep the origin running while you download; press
Ctrl+C in this terminal when finished.
(
GOENV=off GOWORK=off GOTOOLCHAIN=local GOFLAGS= GOOS= GOARCH= go build -o local-origin origin.go &&
./local-origin 127.0.0.1:0
)
Copy the two assignment lines printed by the origin into your first terminal, then set the destination there:
export OUTPUT_PATH='example.bin'
Run the downloader's build-and-run block above. It should exit zero after printing assembly counts
ending at 8388613/8388613 bytes. Verify the actual file independently:
(
printf '%s %s\n' "${EXPECTED_SHA256:?Set EXPECTED_SHA256}" "${OUTPUT_PATH:?Set OUTPUT_PATH}" |
sha256sum --check --status
)
A zero exit status from this check means the saved file matches the fixture digest. For a remote file, obtain the digest through a trusted publisher channel; a checksum from the same untrusted download does not establish authenticity. To try resume behavior on a slower or larger origin, press Ctrl+C during the download and rerun the same command. Only fully saved chunks are reused; this small loopback fixture may finish before you can interrupt it.
Best practices and common pitfalls
This implementation rejects unknown lengths, weak or missing ETags, encoded responses and objects larger than 1 TiB. Empty files are supported when HEAD supplies a strong ETag and zero length. SHA-256 detects corrupt saved chunks and mixed versions even if an origin misbehaves.
Successful runs retain the private .parts directory for explicit cleanup. A hard kill can leave
its .lock directory behind: remove that lock only after confirming no downloader is active.
Keep the output parent and saved chunks under your exclusive control. The lock coordinates this
program's runs using the same parts path; it does not protect against other programs modifying
those files.
The assembled temporary file is created beside the destination so they share a filesystem.
os.Rename replaces the destination on Linux after verification.
This is not a power-loss durability guarantee. A digest mismatch does not repair saved chunks;
choose a new output path and investigate the origin or corrupted state.
Conclusion
Use the local fixture to confirm the complete path, then substitute an origin with compatible range responses and an independently trusted digest. The four-worker downloader retains complete chunks and publishes the assembled file only after verification.
Transloadit’s /http/import Robot handles HTTP imports in processing workflows.