Upload files with Web Workers and bounded streams
Use a worker when your upload pipeline needs file processing away from the page’s main thread.
This example reads a File incrementally, sends one 1 MiB chunk at a time per worker, and waits
for the receiver’s final acknowledgment before showing a download link. Two workers can handle
different files while the remaining files wait in a bounded queue.
Challenges with traditional file uploads
An ordinary asynchronous upload can append a File directly to FormData. You do not need to
read it with FileReader first; see MDN’s FormData examples.
Start there if all you need is to send a file. Workers do not increase network bandwidth, and this
tutorial makes no speed or total-memory claim.
The separate task here is controlling incremental reads and acknowledgments across several files. The worker introduction covers basic worker messages. The parallel chunk tutorial covers concurrent requests within one file; here, requests within each file are sequential.
Meet Web Workers and streams
Web Workers
Workers can run file processing and make requests, but cannot update the page’s DOM. The page
sends each File through structured cloning and renders messages from its worker. It does not
read the file into an ArrayBuffer to send it. See using Web Workers.
JavaScript streams
Blob.stream(), inherited by File,
provides a stream of bytes and is available inside workers. Its read boundaries are not our upload
protocol’s boundaries. The generator below combines those reads into 1 MiB chunks, keeping a
shorter final chunk intact.
Architecture overview
The page admits up to four files, each at most 8 MiB, including empty files. The pool runs at most two jobs. Each worker creates an upload, reads a chunk, waits for its multipart POST to be acknowledged, and finally asks the receiver to complete the upload. Progress counts acknowledged file bytes, not bytes currently in flight.
The receiver is included below. It checks offsets and lengths, retains up to four files in memory, and exposes completed files as downloads. Uploads expire 60 seconds after creation, including completed uploads. Stopping the server loses every file. This is a local protocol demonstration, with no retries, reload recovery, or durable storage.
Set up the local upload demo
Use Node.js 26.8.1 for this local example and a browser supporting module workers, Blob.stream(),
and AbortSignal.timeout(). The complete path was tested on Linux with Chromium 145. Node 26 is a
maintained Current release as of October 1, 2026; check the release schedule
when choosing a runtime for deployment. This receiver is intended for localhost.
In Bash, create a fresh directory. If creation or navigation fails, stop and choose another name before saving files. No dependency installation or enclosing project’s build configuration is needed.
mkdir worker-stream-demo && cd worker-stream-demo
Save the HTML as index.html, put both main-thread blocks in main.ts in the order shown, and
save the worker and receiver as upload-worker.ts and server.mts. The receiver strips TypeScript
annotations when serving the browser scripts; browsers never execute the .ts sources directly.
Main thread (main.js)
Save this as index.html. The scripts will be served from the same local origin.
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title>Worker stream uploads</title>
</head>
<body>
<main>
<h1>Worker stream uploads</h1>
<label for="file-input">Up to four files, each at most 8 MiB</label>
<input id="file-input" type="file" multiple />
<button id="upload-files" type="button">Upload files</button>
<button id="cancel-uploads" type="button" disabled>Cancel uploads</button>
<p id="status" role="status">Choose files.</p>
<ul id="results" aria-label="Upload results"></ul>
</main>
<script type="module" src="/main.js"></script>
</body>
</html>
Start main.ts with this pool. It detaches a finished job before calling the UI, and advances the
queue even if a callback throws. A worker script or message-serialization failure cancels the
whole pool instead of returning a broken worker to service.
export type WorkerReply =
| { type: 'progress'; progress: number; message: string }
| { type: 'complete'; message: string; url: string }
| { type: 'error'; message: string }
type Callbacks = {
onProgress?: (progress: number, message: string) => void
onComplete?: (message: string, url: string) => void
onError?: (message: string) => void
}
type Task = { id: string; file: File }
class WorkerPool {
closed = false
workers: Worker[] = []
idleWorkers: Worker[] = []
taskQueue: Task[] = []
taskCallbacks = new Map<string, Callbacks>()
currentTasks = new Map<Worker, Task>()
constructor(script: string, size = 2) {
if (!Number.isInteger(size) || size < 1 || size > 2) throw new Error('Use one or two workers')
try {
for (let index = 0; index < size; index++) {
const worker = new Worker(script, { type: 'module' })
worker.onmessage = (event: MessageEvent<WorkerReply>) => this.handleWorkerMessage(worker, event.data)
worker.onerror = (event) => {
event.preventDefault()
this.terminate('An upload worker failed. Try again.')
}
worker.onmessageerror = () => this.terminate('An upload worker failed. Try again.')
this.workers.push(worker)
this.idleWorkers.push(worker)
}
} catch (error) {
this.terminate()
throw error
}
}
processFile(file: File, callbacks: Callbacks, id = crypto.randomUUID()): string {
if (this.closed) throw new Error('The upload pool is closed.')
if (this.taskCallbacks.size >= 4) throw new Error('The upload queue is full.')
const task = { id, file }
this.taskCallbacks.set(task.id, callbacks)
const worker = this.idleWorkers.pop()
if (worker) this.runTask(worker, task)
else this.taskQueue.push(task)
return task.id
}
runTask(worker: Worker, task: Task): void {
this.currentTasks.set(worker, task)
try {
worker.postMessage(task)
} catch {
this.terminate('An upload worker failed. Try again.')
}
}
handleWorkerMessage(worker: Worker, data: WorkerReply): void {
if (this.closed) return
const task = this.currentTasks.get(worker)
if (!task) return
const callbacks = this.taskCallbacks.get(task.id)
if (!callbacks) return
if (data.type === 'progress') {
callbacks.onProgress?.(data.progress, data.message)
return
}
this.currentTasks.delete(worker)
this.taskCallbacks.delete(task.id)
try {
if (data.type === 'complete') callbacks.onComplete?.(data.message, data.url)
else callbacks.onError?.(data.message)
} finally {
if (!this.closed) {
const next = this.taskQueue.shift()
if (next) this.runTask(worker, next)
else this.idleWorkers.push(worker)
}
}
}
terminate(message = 'Upload canceled.'): void {
if (this.closed) return
this.closed = true
for (const worker of this.workers) worker.terminate()
this.workers = []
this.idleWorkers = []
this.taskQueue = []
this.currentTasks.clear()
const callbacks = [...this.taskCallbacks.values()]
this.taskCallbacks.clear()
const errors: unknown[] = []
for (const callback of callbacks) {
try {
callback.onError?.(message)
} catch (error) {
errors.push(error)
}
}
if (errors.length > 0) throw new AggregateError(errors, 'Upload cancellation callbacks failed.')
}
}
Append this block to main.ts. File selection and repeated submissions are disabled while a
batch is running or cleaning up. Each row belongs to one file, so a failure cannot be mistaken for
another file’s result. Names are inserted as text, never interpreted as HTML or storage paths.
const input = document.getElementById('file-input')
const upload = document.getElementById('upload-files')
const cancel = document.getElementById('cancel-uploads')
const status = document.getElementById('status')
const results = document.getElementById('results')
if (!(input instanceof HTMLInputElement) || !(upload instanceof HTMLButtonElement) ||
!(cancel instanceof HTMLButtonElement) || !status || !results) throw new Error('Missing controls')
let pool: WorkerPool | null = null
let running = false
cancel.addEventListener('click', () => pool?.terminate())
window.addEventListener('pagehide', () => pool?.terminate())
input.addEventListener('change', () => {
if (!running) status.textContent = 'Ready to upload.'
})
upload.addEventListener('click', async () => {
if (running) return
const files = Array.from(input.files ?? [])
if (files.length < 1 || files.length > 4 || files.some((file) => file.size > 8 * 1024 * 1024)) {
status.textContent = 'Choose one to four files, each at most 8 MiB.'
return
}
running = true
input.disabled = upload.disabled = true
cancel.disabled = false
results.replaceChildren()
status.textContent = 'Uploading…'
const failedIds: string[] = []
let failures = 0
try {
const batchPool = new WorkerPool('/upload-worker.js')
pool = batchPool
const jobs = files.map((file) => new Promise<void>((resolve) => {
const id = crypto.randomUUID()
const row = document.createElement('li')
const name = document.createElement('p')
name.textContent = file.name
const progress = document.createElement('progress')
progress.max = 100
progress.value = 0
progress.setAttribute('aria-label', `Acknowledged bytes for ${file.name}`)
const message = document.createElement('p')
message.setAttribute('role', 'status')
message.textContent = 'Queued.'
row.append(name, progress, message)
results.append(row)
batchPool.processFile(file, {
onProgress(value, text) { progress.value = value; message.textContent = text },
onComplete(text, url) {
progress.value = 100
message.textContent = text
const link = document.createElement('a')
link.href = url
link.download = 'upload.bin'
link.textContent = `Download ${file.name}`
row.append(link)
resolve()
},
onError(text) {
failures++
failedIds.push(id)
message.textContent = text
resolve()
},
}, id)
}))
await Promise.all(jobs)
status.textContent = failures === 0 ? 'All uploads complete.' : 'Some uploads did not complete.'
} catch {
status.textContent = 'Could not start the upload workers. Try again.'
} finally {
pool?.terminate()
pool = null
let cleanupFailed = false
for (const id of failedIds) {
try {
const response = await fetch(`/uploads/${id}`, {
method: 'DELETE', headers: { 'X-Upload-Demo': '1' }, signal: AbortSignal.timeout(5000),
})
if (!response.ok) cleanupFailed = true
} catch { cleanupFailed = true }
}
if (cleanupFailed) status.textContent += ' Cleanup was not confirmed; uploads will expire.'
running = false
input.disabled = upload.disabled = false
cancel.disabled = true
}
})
Worker implementation (upload-worker.js)
Save this as upload-worker.ts. Awaiting each POST prevents the generator from producing another
application chunk while the previous request is pending. The browser may still buffer stream reads
and request data internally. The 1 MiB chunk size is not a bound on the browser’s total memory.
import type { WorkerReply } from './main.js'
declare const self: DedicatedWorkerGlobalScope
const CHUNK_SIZE = 1024 * 1024
async function* readChunks(file: Blob): AsyncGenerator<Uint8Array<ArrayBuffer>> {
const reader = file.stream().getReader()
let buffer = new Uint8Array(CHUNK_SIZE)
let used = 0
let finished = false
try {
while (true) {
const { done, value } = await reader.read()
if (done) { finished = true; break }
let offset = 0
while (offset < value.length) {
const length = Math.min(CHUNK_SIZE - used, value.length - offset)
buffer.set(value.subarray(offset, offset + length), used)
used += length
offset += length
if (used === CHUNK_SIZE) {
yield buffer
buffer = new Uint8Array(CHUNK_SIZE)
used = 0
}
}
}
if (used > 0) yield buffer.subarray(0, used)
} finally {
if (!finished) await reader.cancel().catch(() => {})
reader.releaseLock()
}
}
function send(message: WorkerReply): void { self.postMessage(message) }
async function request(path: string, options: RequestInit): Promise<Response> {
const response = await fetch(path, {
...options, headers: { ...options.headers, 'X-Upload-Demo': '1' },
signal: AbortSignal.timeout(5000),
})
if (!response.ok) throw new Error('Upload request rejected')
return response
}
self.onmessage = async (event: MessageEvent<{ file: File; id: string }>) => {
const { file, id } = event.data
try {
if (!(file instanceof File) || file.size > 8 * CHUNK_SIZE) throw new Error('Invalid file')
await request(`/uploads/${id}`, { method: 'POST', headers: { 'Upload-Length': String(file.size) } })
send({ type: 'progress', progress: 0, message: 'Uploading…' })
let acknowledged = 0
for await (const chunk of readChunks(file)) {
const form = new FormData()
form.append('offset', String(acknowledged))
form.append('chunk', new Blob([chunk]), 'chunk.bin')
await request(`/uploads/${id}/chunks`, { method: 'POST', body: form })
acknowledged += chunk.length
send({ type: 'progress', progress: 100 * acknowledged / file.size,
message: `${acknowledged} / ${file.size} bytes acknowledged.` })
}
send({ type: 'progress', progress: 100, message: 'Confirming…' })
const response = await request(`/uploads/${id}/complete`, { method: 'POST' })
if (await response.text() !== String(file.size)) throw new Error('Unexpected acknowledgment')
send({ type: 'complete', message: 'Upload complete.', url: `/uploads/${id}/file` })
} catch {
send({ type: 'error', message: 'Upload failed. Try again.' })
}
}
This uses discrete multipart requests, not a ReadableStream request body. Let the browser set
the multipart Content-Type boundary. An empty file sends no chunks but still requires completion.
Why not read the whole file once in the worker?
Moving a whole-file arrayBuffer() call into a worker still creates a whole-file application
buffer. Here, the coalescer retains a 1 MiB buffer plus the current stream read, and each active
job has only one chunk request awaiting acknowledgment. A Blob and the networking implementation
may create additional copies. Queued jobs hold file handles rather than pre-read file bytes.
Building a tiny worker pool
The pool above has two workers and at most four pending jobs. This is a scheduling limit, not a recommendation based on CPU count. More workers can increase processing, memory, and request overhead without increasing useful throughput. Measure against your own processing step and receiver.
Receive the chunks locally
Save this as server.mts. The .mts extension keeps execution unambiguously in ES-module mode,
including inside a CommonJS project. stripTypeScriptTypes()
removes annotations; it does not type-check the scripts.
import { readFile } from 'node:fs/promises'
import { createServer } from 'node:http'
import type { IncomingMessage, ServerResponse } from 'node:http'
import { stripTypeScriptTypes } from 'node:module'
const CHUNK_SIZE = 1024 * 1024
const MAX_BODY = CHUNK_SIZE + 4096
const TTL_MS = 60_000
type Upload = { bytes: Buffer; received: number; complete: boolean; expires: number }
const uploads = new Map<string, Upload>()
class HttpError extends Error {
status: number
constructor(status: number, message: string) { super(message); this.status = status }
}
function expire(): void {
for (const [id, upload] of uploads) {
if (upload.expires <= Date.now()) uploads.delete(id)
}
}
async function readBody(req: IncomingMessage, limit: number): Promise<Buffer> {
const bytes = Buffer.alloc(limit)
let length = 0
// Keep the connection long enough to report an oversize body.
for await (const part of req.iterator({ destroyOnReturn: false })) {
if (!Buffer.isBuffer(part)) throw new HttpError(400, 'Expected bytes')
if (length + part.length > limit) throw new HttpError(413, 'Body too large')
part.copy(bytes, length)
length += part.length
}
return bytes.subarray(0, length)
}
async function main(): Promise<void> {
const assets = new Map<string, string>([['/', await readFile(new URL('./index.html', import.meta.url), 'utf8')]])
for (const name of ['main', 'upload-worker']) {
assets.set(`/${name}.js`, stripTypeScriptTypes(await readFile(new URL(`./${name}.ts`, import.meta.url), 'utf8')))
}
const port = Number(process.argv[2] ?? 0)
if (!Number.isInteger(port) || port < 0 || port > 65535) throw new Error('Invalid port')
let origin = ''
let active = 0
async function handle(req: IncomingMessage, res: ServerResponse): Promise<void> {
if (req.headers.host !== new URL(origin).host ||
(req.headers.origin !== undefined && req.headers.origin !== origin)) throw new HttpError(403, 'Use the printed local URL')
const method = req.method
if (method !== 'GET' && req.headers['x-upload-demo'] !== '1') throw new HttpError(403, 'Missing demo header')
const path = new URL(req.url ?? '/', origin).pathname
const isChunk = method === 'POST' && path.endsWith('/chunks')
const body = await readBody(req, isChunk ? MAX_BODY : 0)
// Parse before looking up an upload: pending work cannot retain a deleted session.
let chunk: Uint8Array | undefined
let offset: string | undefined
if (isChunk) {
const contentType = req.headers['content-type']
if (typeof contentType !== 'string' || !contentType.startsWith('multipart/form-data;')) throw new HttpError(400, 'Expected multipart data')
let form: FormData
try {
form = await new Request(origin, { method: 'POST', headers: { 'Content-Type': contentType }, body: new Uint8Array(body) }).formData()
} catch { throw new HttpError(400, 'Invalid multipart data') }
const value = form.get('chunk')
const position = form.get('offset')
if (!(value instanceof Blob) || value.size > CHUNK_SIZE || typeof position !== 'string' ||
!/^(0|[1-9]\d*)$/.test(position) || [...form.keys()].length !== 2) throw new HttpError(400, 'Invalid chunk fields')
chunk = new Uint8Array(await value.arrayBuffer())
offset = position
}
expire()
res.setHeader('Cache-Control', 'no-store')
res.setHeader('X-Content-Type-Options', 'nosniff')
const asset = assets.get(path)
if (method === 'GET' && asset !== undefined) {
res.setHeader('Content-Type', path === '/' ? 'text/html; charset=utf-8' : 'text/javascript')
res.end(asset)
return
}
const match = /^\/uploads\/([a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12})(?:\/(chunks|complete|file))?$/.exec(path)
if (!match) throw new HttpError(404, 'Unknown route')
const [, id, operation] = match
if (method === 'POST' && operation === undefined) {
const size = req.headers['upload-length']
if (typeof size !== 'string' || !/^(0|[1-9]\d*)$/.test(size) || Number(size) > 8 * CHUNK_SIZE) throw new HttpError(400, 'Expected a size from 0 to 8 MiB')
if (uploads.has(id)) throw new HttpError(409, 'Upload already exists')
if (uploads.size >= 4) throw new HttpError(503, 'Upload capacity reached')
uploads.set(id, { bytes: Buffer.alloc(Number(size)), received: 0, complete: false, expires: Date.now() + TTL_MS })
res.writeHead(201).end()
return
}
if (method === 'DELETE' && operation === undefined) {
uploads.delete(id)
res.writeHead(204).end()
return
}
const upload = uploads.get(id)
if (!upload) throw new HttpError(404, 'Upload missing or expired')
if (isChunk && operation === 'chunks') {
const expected = Math.min(CHUNK_SIZE, upload.bytes.length - upload.received)
if (!chunk || chunk.length === 0 || chunk.length !== expected || offset !== String(upload.received) || upload.complete) throw new HttpError(409, 'Unexpected chunk offset or length')
upload.bytes.set(chunk, upload.received)
upload.received += chunk.length
res.writeHead(204).end()
return
}
if (method === 'POST' && operation === 'complete') {
if (upload.received !== upload.bytes.length) throw new HttpError(409, 'Missing bytes')
upload.complete = true
res.end(String(upload.received))
return
}
if (method === 'GET' && operation === 'file') {
if (!upload.complete) throw new HttpError(409, 'Upload is not complete')
res.setHeader('Content-Type', 'application/octet-stream')
res.setHeader('Content-Disposition', 'attachment; filename="upload.bin"')
res.end(upload.bytes)
return
}
throw new HttpError(405, 'Unsupported operation')
}
const server = createServer({ requestTimeout: 10_000, headersTimeout: 10_000 }, (req, res) => {
if (active >= 6) { res.writeHead(503, { Connection: 'close' }).end('Too many requests'); return }
active++
let handled = false
let closed = false
const deadline = setTimeout(() => { req.destroy(); res.destroy() }, 10_000)
function release(): void {
if (handled && closed) { clearTimeout(deadline); active-- }
}
res.once('close', () => { closed = true; release() })
handle(req, res).catch((error: unknown) => {
const status = error instanceof HttpError ? error.status : 500
const message = error instanceof HttpError ? error.message : 'Request failed'
if (!res.destroyed) res.writeHead(status, { Connection: 'close' }).end(message)
}).finally(() => { handled = true; release() })
})
server.maxConnections = 16
await new Promise<void>((resolve, reject) => {
server.once('error', reject)
server.listen(port, '127.0.0.1', () => resolve())
})
const address = server.address()
if (!address || typeof address === 'string') throw new Error('Missing server address')
origin = `http://127.0.0.1:${address.port}`
setInterval(expire, 1000).unref()
process.once('SIGINT', () => {
server.close()
server.closeAllConnections()
})
console.log(`Open ${origin}`)
}
main().catch((error: unknown) => {
console.error(error instanceof Error ? error.message : 'Could not start the server')
process.exitCode = 1
})
The four registered file buffers total at most 32 MiB. Request parsing uses additional buffers; each accepted body is limited to 1 MiB plus 4 KiB of multipart overhead, including trailing data. There are at most six active handlers. A download can retain its file buffer after deletion or expiry, so the handler’s slot stays occupied until both handling and response closure finish. These limits describe application operations, not total process memory or garbage-collection timing.
From the directory with the four saved files, run this foreground command:
node server.mts
Open the printed http://127.0.0.1:PORT URL. Select files and click
Upload files. Each row reaches
Confirming… after its chunks are acknowledged; only a successful
completion response produces Upload complete. and a download link.
Click the link to save the received bytes as upload.bin through your browser. A matching byte
count is not an integrity checksum; compare the downloaded file with the original when checking
this example.
Stop the server with Ctrl+C, which closes its connections. To select a particular free port, append
its number to node server.mts.
An occupied port or missing source file exits with an error rather than printing a ready URL.
Browser compatibility
Check the browser versions you support for Worker, Blob.stream(), and
AbortSignal.timeout().
This example does not transfer a stream between threads. Its five-second browser request timeout
uses active time and may pause in a suspended worker; the receiver has a separate ten-second
deadline. Other browsers and operating systems were not exercised for this walkthrough.
Memory management best practices
Keep one application chunk in flight per worker and a small queue of file handles. The generator
cancels an unfinished stream and releases its reader lock when a request fails. The page terminates
all workers at the end of a batch, on cancellation, and on pagehide.
Click Cancel uploads while a batch is pending. Active and queued
rows receive Upload canceled.; already completed rows retain
their results. The page attempts to delete failed or canceled upload IDs and holds the controls
disabled until cleanup settles. A lost creation response, a page closing, or a failed deletion can
leave bytes until expiry. Terminating a worker does not run its finally blocks or undo a request
already accepted by the receiver.
Security and resilience
The server binds to 127.0.0.1, checks the host and browser origin, and requires a custom header for
mutations. It does not enable cross-origin access, interpret filenames as paths, or serve uploaded
bytes as executable HTML. These are local demo boundaries, not authentication or content validation.
Do not expose this receiver as a public upload service.
A deployed service needs authentication, authorization, per-user quotas, HTTPS, durable storage, and validation appropriate to its consumers. This offset protocol deliberately rejects duplicate or reordered chunks. Adding retries requires an idempotency contract; reload recovery requires persisted state and reconciliation with the receiver.
Debugging Web Workers
Inspect worker requests and messages in browser developer tools. A script-loading error cancels the pool; an HTTP failure becomes Upload failed. Try again. in the corresponding row. A 409 response indicates a wrong offset, wrong chunk length, or incomplete file. A 503 can mean all four upload slots are occupied, including completed files; wait for their expiry before retrying. Missing or expired IDs return 404.
Common pitfalls
- Treating 100% acknowledged bytes as completion. The last POST can still fail.
- Re-enabling input before canceled work settles. This page waits for its cleanup attempt.
- Assuming stream read sizes match HTTP chunk sizes. The coalescer handles a shorter remainder.
- Assuming a terminated worker deletes server state. Expiry remains necessary.
- Reading every queued file in advance, which defeats the incremental processing path.
Key takeaways
Try an empty file and a binary file slightly larger than 1 MiB, then compare each downloaded result with its input. The useful boundary is the complete path from an incremental read to an acknowledged chunk and a confirmed file, with a small number of active jobs. For production resumability, start with a maintained protocol such as tus and a compatible receiver. Uppy’s tus plugin supplies a maintained browser client.