Optimizing online file uploads with chunking and parallel uploads

Kevin van Zonneveld

Co-founder · Amsterdam, The Netherlands ·

To upload chunks in parallel, give each chunk a stable position and publish the file only after the receiver has checked the complete result. Here you will build both sides: a browser sends three chunks at a time, and a local Node.js server makes the assembled file available for download after verifying its SHA-256 digest.

Choose a small, reproducible upload

This example is for developers learning how parallel chunk uploads fit together. Use Node.js 24.15.0 or a later maintained release and a current Chromium browser; the example was tested on Linux with Node.js 24.15.0 and Chromium 145. Node.js 24 is an LTS release. There are no packages to install.

The page accepts a nonempty JPEG, PNG, or PDF up to 8 MiB. That small limit is deliberate: the receiver stores files in memory, and the browser hashes whole files with crypto.subtle.digest(), which does not accept streaming input. This is a local protocol lesson, not a large-file storage service. Restarting the server loses every file. Reloading the page loses the client's upload state.

Give each chunk a position

Use 256 KiB chunks, numbered from zero. A file of 524,295 bytes has three chunks: two of 262,144 bytes and one of seven bytes. Blob.slice(start, end) excludes the end position, so adjacent slices neither overlap nor leave gaps.

The protocol has five operations:

POST /uploads reserves a file size and SHA-256 digest and returns a server-generated ID.
PUT /uploads/:id/:index writes the raw chunk at its numbered position. An identical retry succeeds; a different body at an already accepted position fails with HTTP 409.
POST /uploads/:id/complete checks that every position is present and the whole-file digest matches. Repeating this operation returns the same digest.
GET /uploads/:id/file returns bytes only after completion. The browser checks these bytes too.
DELETE /uploads/:id removes the upload, including a completed file.

Arrival order does not determine file order. A lost acknowledgment can cause a duplicate request, which is why chunk writes and completion must be idempotent. Creation is not retried automatically: a lost creation response leaves an unknown ID, which the server eventually expires.

Create the page

Create a new empty directory and save the following three files side by side. Save this first file as index.html:

<!doctype html>
<html lang="en">
  <head>
    <meta charset="utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1" />
    <meta name="color-scheme" content="light dark" />
    <title>Parallel chunk upload</title>
  </head>
  <body>
    <main>
      <h1>Parallel chunk upload</h1>
      <label for="file">JPEG, PNG, or PDF, up to 8 MiB</label>
      <input id="file" type="file" accept="image/jpeg,image/png,application/pdf" />
      <button id="upload" type="button">Upload</button>
      <button id="cancel" type="button" disabled>Cancel</button>
      <p id="status" role="status">Choose a file.</p>
      <a id="download" hidden download="upload.bin">Download verified file</a>
    </main>
    <script type="module" src="/client.js"></script>
  </body>
</html>

Receive and publish the chunks

Save this as server.mts. The .mts extension makes it an ES module even inside a CommonJS project. The server uses Node's stripTypeScriptTypes() to serve the next file as JavaScript. In Node.js 24.15.0 that API emits an experimental warning.

Four uploads may exist at once, including completed uploads. Each expires 60 seconds after creation, even if requests are still arriving. Six requests may be active, each with a 10-second deadline. Incoming bodies are read before looking up session data, so a pending body cannot keep a deleted session alive or write into it later.

import { createHash, randomUUID } from 'node:crypto'
import { readFile } from 'node:fs/promises'
import { createServer } from 'node:http'
import type { IncomingMessage, ServerResponse } from 'node:http'
import { stripTypeScriptTypes } from 'node:module'

const CHUNK_SIZE = 256 * 1024
const MAX_FILE_SIZE = 8 * 1024 * 1024
const MAX_UPLOADS = 4
const MAX_REQUESTS = 6
const TTL_MS = 60_000

type Upload = {
  bytes: Buffer
  digest: string
  seen: Set<number>
  complete: boolean
  expires: number
}
const uploads = new Map<string, Upload>()

class HttpError extends Error {
  status: number
  constructor(status: number, message: string) {
    super(message)
    this.status = status
  }
}

function expire(): void {
  for (const [id, upload] of uploads) {
    if (upload.expires <= Date.now()) uploads.delete(id)
  }
}

async function readBody(req: IncomingMessage, limit: number): Promise<Buffer> {
  const bytes = Buffer.alloc(limit)
  let length = 0
  // Leave the socket open long enough to send a useful error response.
  for await (const part of req.iterator({ destroyOnReturn: false })) {
    if (!Buffer.isBuffer(part)) throw new HttpError(400, 'Expected bytes')
    if (length + part.length > limit) throw new HttpError(413, 'Body too large')
    part.copy(bytes, length)
    length += part.length
  }
  return bytes.subarray(0, length)
}

async function main(): Promise<void> {
  const html = await readFile(new URL('./index.html', import.meta.url))
  const client = stripTypeScriptTypes(
    await readFile(new URL('./client.ts', import.meta.url), 'utf8'),
  )
  const port = Number(process.argv[2] ?? 0)
  if (!Number.isInteger(port) || port < 0 || port > 65535) throw new Error('Invalid port')
  let origin = ''
  let active = 0

  async function handle(req: IncomingMessage, res: ServerResponse): Promise<void> {
    if (req.headers.host !== new URL(origin).host ||
        (req.headers.origin !== undefined && req.headers.origin !== origin)) {
      throw new HttpError(403, 'Use the printed local URL')
    }
    const method = req.method
    if (method !== 'GET' && req.headers['x-upload-demo'] !== '1') {
      throw new HttpError(403, 'Missing demo header')
    }
    const path = new URL(req.url ?? '/', origin).pathname
    const body = await readBody(req, method === 'PUT' ? CHUNK_SIZE : 0)
    expire()
    res.setHeader('Cache-Control', 'no-store')
    res.setHeader('X-Content-Type-Options', 'nosniff')

    if (method === 'GET' && (path === '/' || path === '/client.js')) {
      res.setHeader('Content-Type', path === '/' ? 'text/html; charset=utf-8' : 'text/javascript')
      res.end(path === '/' ? html : client)
      return
    }
    if (method === 'POST' && path === '/uploads') {
      const length = req.headers['upload-length']
      const digest = req.headers['upload-sha256']
      const size = Number(length)
      if (typeof length !== 'string' || !/^[1-9]\d*$/.test(length) ||
          !Number.isSafeInteger(size) || size > MAX_FILE_SIZE) {
        throw new HttpError(400, 'File must be between 1 byte and 8 MiB')
      }
      if (typeof digest !== 'string' || !/^[a-f0-9]{64}$/.test(digest)) {
        throw new HttpError(400, 'Expected a SHA-256 digest')
      }
      if (uploads.size >= MAX_UPLOADS) throw new HttpError(503, 'Upload capacity reached')
      const id = randomUUID()
      uploads.set(id, {
        bytes: Buffer.alloc(size), digest, seen: new Set(), complete: false,
        expires: Date.now() + TTL_MS,
      })
      res.writeHead(201).end(id)
      return
    }

    const match = /^\/uploads\/([a-f0-9-]{36})(?:\/(\d+|complete|file))?$/.exec(path)
    if (!match) throw new HttpError(404, 'Unknown route')
    const [, id, operation] = match
    if (method === 'DELETE' && operation === undefined) {
      uploads.delete(id)
      res.writeHead(204).end()
      return
    }
    const upload = uploads.get(id)
    if (!upload) throw new HttpError(404, 'Upload missing or expired')
    const count = Math.ceil(upload.bytes.length / CHUNK_SIZE)

    if (method === 'PUT' && operation !== undefined && /^\d+$/.test(operation)) {
      const index = Number(operation)
      if (!Number.isSafeInteger(index) || index >= count) {
        throw new HttpError(400, 'Invalid chunk index')
      }
      const start = index * CHUNK_SIZE
      const target = upload.bytes.subarray(start, Math.min(start + CHUNK_SIZE, upload.bytes.length))
      if (body.length !== target.length) throw new HttpError(400, 'Wrong chunk length')
      if (upload.seen.has(index)) {
        if (!body.equals(target)) throw new HttpError(409, 'Conflicting chunk')
      } else {
        body.copy(target)
        upload.seen.add(index)
      }
      res.writeHead(204).end()
      return
    }
    if (method === 'POST' && operation === 'complete') {
      if (upload.seen.size !== count) throw new HttpError(409, 'Missing chunks')
      if (createHash('sha256').update(upload.bytes).digest('hex') !== upload.digest) {
        throw new HttpError(422, 'Digest mismatch')
      }
      upload.complete = true
      res.end(upload.digest)
      return
    }
    if (method === 'GET' && operation === 'file') {
      if (!upload.complete) throw new HttpError(409, 'Upload is not complete')
      res.setHeader('Content-Type', 'application/octet-stream')
      res.setHeader('Content-Disposition', 'attachment; filename="upload.bin"')
      res.end(upload.bytes)
      return
    }
    throw new HttpError(405, 'Unsupported operation')
  }

  const server = createServer({ requestTimeout: 10_000, headersTimeout: 10_000 }, (req, res) => {
    if (active >= MAX_REQUESTS) {
      res.writeHead(503, { Connection: 'close' }).end('Too many requests')
      return
    }
    active++
    let handled = false
    let closed = false
    const deadline = setTimeout(() => { req.destroy(); res.destroy() }, 10_000)
    function release(): void {
      if (handled && closed) { clearTimeout(deadline); active-- }
    }
    res.once('close', () => { closed = true; release() })
    handle(req, res).catch((error: unknown) => {
      const status = error instanceof HttpError ? error.status : 500
      const message = error instanceof HttpError ? error.message : 'Request failed'
      if (!res.destroyed) res.writeHead(status, { Connection: 'close' }).end(message)
    }).finally(() => { handled = true; release() })
  })
  server.maxConnections = 16
  await new Promise<void>((resolve, reject) => {
    server.once('error', reject)
    server.listen(port, '127.0.0.1', () => resolve())
  })
  const address = server.address()
  if (!address || typeof address === 'string') throw new Error('Missing server address')
  origin = `http://127.0.0.1:${address.port}`
  setInterval(expire, 1000).unref()
  console.log(`Open ${origin}`)
}

main().catch((error: unknown) => {
  console.error(error instanceof Error ? error.message : 'Could not start the server')
  process.exitCode = 1
})

The four session buffers total at most 32 MiB. Separately, an admitted request can retain a 256 KiB incoming buffer or reference an 8 MiB outgoing file until its response closes. Deletion and expiry do not release that request's slot early. These are application buffer limits, not a bound on Node's total memory usage or garbage-collection timing. Header timeouts and the connection limit also bound this local server's waiting connections; see the Node.js HTTP documentation.

Send at most three chunks at once

Save this as client.ts. Each batch waits for all its requests to settle before another batch starts. A slow chunk therefore holds up its batch, but the limit is easy to inspect and retries cannot multiply the number of active chunk requests.

const CHUNK_SIZE = 256 * 1024
const MAX_FILE_SIZE = 8 * 1024 * 1024
const CONCURRENCY = 3

const input = document.getElementById('file')
const uploadButton = document.getElementById('upload')
const cancelButton = document.getElementById('cancel')
const status = document.getElementById('status')
const download = document.getElementById('download')
if (!(input instanceof HTMLInputElement) || !(uploadButton instanceof HTMLButtonElement) ||
    !(cancelButton instanceof HTMLButtonElement) || !(status instanceof HTMLParagraphElement) ||
    !(download instanceof HTMLAnchorElement)) throw new Error('Missing upload controls')

class HttpError extends Error {
  status: number
  constructor(status: number) { super(`HTTP ${status}`); this.status = status }
}

async function validateFile(file: File): Promise<void> {
  if (file.size === 0 || file.size > MAX_FILE_SIZE) throw new Error('Choose a file between 1 byte and 8 MiB.')
  const signatures: Record<string, number[]> = {
    'image/jpeg': [0xff, 0xd8, 0xff],
    'image/png': [0x89, 0x50, 0x4e, 0x47],
    'application/pdf': [0x25, 0x50, 0x44, 0x46],
  }
  const expected = signatures[file.type]
  if (!expected) throw new Error('Choose a JPEG, PNG, or PDF.')
  const header = new Uint8Array(await file.slice(0, 4).arrayBuffer())
  if (!expected.every((byte, index) => header[index] === byte)) throw new Error('Invalid file signature.')
}

async function sha256(blob: Blob): Promise<string> {
  const hash = await crypto.subtle.digest('SHA-256', await blob.arrayBuffer())
  return Array.from(new Uint8Array(hash), (byte) => byte.toString(16).padStart(2, '0')).join('')
}

function waitForRetry(ms: number, signal: AbortSignal): Promise<void> {
  signal.throwIfAborted()
  return new Promise((resolve, reject) => {
    const onAbort = () => { clearTimeout(timer); reject(signal.reason) }
    const timer = setTimeout(() => {
      signal.removeEventListener('abort', onAbort)
      resolve()
    }, ms)
    signal.addEventListener('abort', onAbort, { once: true })
  })
}

async function request(
  path: string, options: RequestInit, signal: AbortSignal, retry = true,
): Promise<Blob> {
  for (let attempt = 0; ; attempt++) {
    signal.throwIfAborted()
    try {
      const response = await fetch(path, {
        ...options,
        signal: AbortSignal.any([signal, AbortSignal.timeout(5000)]),
        headers: { ...options.headers, 'X-Upload-Demo': '1' },
      })
      if (!response.ok) throw new HttpError(response.status)
      return await response.blob()
    } catch (error) {
      signal.throwIfAborted()
      if (!retry || attempt === 2 ||
          (error instanceof HttpError && ![408, 429, 500, 502, 503, 504].includes(error.status))) {
        throw error
      }
      await waitForRetry(250 * 2 ** attempt, signal)
    }
  }
}

let running: AbortController | null = null
cancelButton.addEventListener('click', () => running?.abort())
input.addEventListener('change', () => {
  if (running) return
  download.hidden = true
  status.textContent = 'Ready to upload.'
})

uploadButton.addEventListener('click', async () => {
  if (running) return
  const file = input.files?.[0]
  if (!file) { status.textContent = 'Choose a file.'; return }
  const controller = new AbortController()
  const signal = controller.signal
  running = controller
  input.disabled = uploadButton.disabled = true
  cancelButton.disabled = false
  download.hidden = true
  let id: string | undefined
  let verified = false
  try {
    status.textContent = 'Checking file…'
    await validateFile(file)
    const digest = await sha256(file)
    signal.throwIfAborted()
    // Finish creation so cancellation can learn the ID and delete it.
    id = await (await request('/uploads', {
      method: 'POST', headers: { 'Upload-Length': String(file.size), 'Upload-SHA256': digest },
    }, new AbortController().signal, false)).text()
    signal.throwIfAborted()
    const count = Math.ceil(file.size / CHUNK_SIZE)
    let acknowledged = 0
    for (let first = 0; first < count; first += CONCURRENCY) {
      signal.throwIfAborted()
      const batch = []
      for (let index = first; index < Math.min(first + CONCURRENCY, count); index++) {
        const chunk = file.slice(index * CHUNK_SIZE, (index + 1) * CHUNK_SIZE)
        batch.push(request(`/uploads/${id}/${index}`, { method: 'PUT', body: chunk }, signal).then(() => {
          signal.throwIfAborted()
          acknowledged += chunk.size
          status.textContent = `${Math.round(100 * acknowledged / file.size)}% of bytes acknowledged.`
        }))
      }
      const results = await Promise.allSettled(batch)
      const failure = results.find((result) => result.status === 'rejected')
      if (failure) throw failure.reason
    }
    status.textContent = 'All chunks acknowledged. Verifying…'
    const confirmation = await request(`/uploads/${id}/complete`, { method: 'POST' }, signal)
    if (await confirmation.text() !== digest) throw new Error('Unexpected confirmation.')
    const result = await request(`/uploads/${id}/file`, {}, signal)
    if (result.size !== file.size || await sha256(result) !== digest) throw new Error('Downloaded bytes differ.')
    signal.throwIfAborted()
    verified = true
    download.href = `/uploads/${id}/file`
    download.hidden = false
    status.textContent = 'Upload verified. Download is available until the upload expires.'
  } catch (error) {
    status.textContent = signal.aborted ? 'Upload canceled.' :
      `Upload failed: ${error instanceof Error ? error.message : 'Please try again.'}`
  } finally {
    if (id && !verified) {
      try {
        await request(`/uploads/${id}`, { method: 'DELETE' }, new AbortController().signal)
      } catch {
        status.textContent += ' Cleanup could not be confirmed; the server will expire the upload.'
      }
    }
    running = null
    input.disabled = uploadButton.disabled = false
    cancelButton.disabled = true
  }
})

The percentage counts acknowledged bytes, including a shorter last chunk. It does not measure bytes currently in flight. Even 100% is not success: completion and the subsequent download verification must succeed before the link appears. More parallel requests can improve throughput when one request leaves capacity unused, but they also add overhead. Measure against your own receiver and network; this demo makes no speed claim.

Run and interrupt an upload

From the directory containing the three saved files, run:

node server.mts

Open the printed http://127.0.0.1:PORT URL. Do not open index.html directly. Select a file and click Upload. After All chunks acknowledged. Verifying…, the Download verified file link appears. The page has fetched and checked the server's file; clicking the link starts a separate download, whose save location is controlled by your browser. The server always suggests upload.bin as the downloaded filename.

While work is pending, the file input and upload button are disabled. Click Cancel to abort requests and retry waits. Creation and local hashing finish before cancellation is observed; cleanup then attempts to delete the known ID. The controls remain disabled until cleanup settles. A failed cleanup or a lost creation response may leave data until expiry. Canceling a request cannot undo a completion already processed by the server, so the cleanup deletes completed uploads too.

AbortSignal.any() and AbortSignal.timeout() combine user cancellation with a five-second timeout per request attempt. Network failures and the listed temporary HTTP statuses get at most three attempts, with 250 ms and 500 ms retry delays. Other HTTP errors fail immediately. These browser timeouts use active time and can pause while the page is suspended; the server's deadline and expiry are independent.

A 409 response means chunks are missing or a duplicate conflicts. A 422 means the assembled digest is wrong. A 503 can mean all four upload slots are occupied, including completed files; wait for expiry and start again. Missing or expired IDs return 404. Stop the server with Ctrl+C when finished. To use a specific free port, pass its number after server.mts; an occupied port exits with an error instead of printing a ready URL.

Keep the local boundary explicit

The receiver binds only to 127.0.0.1, checks the host and browser origin, requires a custom header for mutations, and never uses a supplied filename as a path. It serves opaque bytes as attachments. The client signature check catches a mistaken file selection; neither a matching prefix nor a matching digest proves that a file is safe. The receiver independently enforces lengths, positions, capacity, and the digest, but does not validate image/PDF structure or scan for malware.

Do not expose this server as a public upload service. A deployed receiver needs authentication, per-user authorization and quotas, HTTPS, durable storage, and content validation appropriate to its consumers. The IDs here isolate local uploads; they are not an account permission system.

Choose the next step

Try a file slightly larger than two chunks and watch the requests in your browser's network panel. The last chunk should be smaller, and the file URL should become usable only after completion. That boundary is the part to retain when replacing memory with durable storage.

For reload recovery, use a maintained protocol and persist enough state to reconcile with the receiver. tus defines offset discovery with HEAD and resumption with PATCH; parallel partial uploads use its optional concatenation extension. This numbered-chunk demo is a separate protocol, not a tus client. Uppy's tus plugin is a practical next step when you want a maintained browser client for a compatible tus server.