Boost JS file uploads using Web Workers and streams

Kevin van Zonneveld

Co-founder · Amsterdam, The Netherlands · Show bio

Uploading files efficiently is crucial for modern web applications. Traditional approaches often read entire files into memory, which can freeze the UI, consume excessive RAM, and lead to a poor user experience, especially with large files. By offloading heavy processing to Web Workers and streaming data in manageable chunks using JavaScript Streams, you can maintain a responsive main thread—even when users upload multi-gigabyte files.

Challenges with traditional file uploads

A common, yet inefficient, approach involves:

1. Reading the entire File object into memory using FileReader.
2. Constructing a FormData object with the file data.
3. Sending the FormData via a fetch or XMLHttpRequest POST request.

Reading whole files into application buffers can cause large memory spikes. Appending a File directly to FormData does not require that read: ordinary uploads are already asynchronous. Workers help when CPU-heavy processing is also needed, not by increasing network bandwidth.

Meet Web Workers and streams

Web Workers

Web Workers allow you to run JavaScript code in background threads, separate from the main execution thread that handles the UI. This means computationally intensive tasks like hashing, compression, or chunking data for uploads won't block rendering, scrolling, or user input, resulting in a smoother experience during file uploads.

JavaScript streams

The Streams API enables processing data incrementally as chunks. Instead of loading an entire file into memory, you can read and process small pieces (typically Uint8Arrays) as they become available. This drastically reduces memory usage and allows data to be sent over the network almost immediately after being read, making JavaScript streams useful for large uploads.

Architecture overview

A parallel upload system using these technologies typically involves:

1. Main Thread: Handles UI interactions (like drag-and-drop), file selection, and displaying progress updates. It passes File objects to the worker pool.
2. Worker Pool: A set of Web Workers manages the file processing tasks. Each available worker receives a File reference.
3. Individual Worker: Uses the Blob.stream() or File.stream() API to read the file chunk by chunk. Each chunk is then POSTed to the back-end upload endpoint. Progress messages (percentage complete) and status updates (completion, errors) are sent back to the main thread.
4. Back-end: Receives the chunks and reassembles them into the complete file. This often involves protocols like tus, cloud storage multipart uploads (e.g., S3 Multipart Upload), or custom server-side logic.

Streaming a file without freezing the UI

The following examples demonstrate a minimal but practical pattern for chunked uploads using a worker pool. Note the inclusion of error handling and cleanup mechanisms.

Main thread (main.js)

This script sets up the worker pool and handles file input events, delegating the processing of each file to the pool. Put the WorkerPool definition later in this article before this code in main.js, and load that file after this HTML. Serve both scripts from the same origin over localhost or HTTPS. The callbacks below log progress; a production interface should render it visibly.

<label for="file-input">Files to upload</label>
<input type="file" id="file-input" multiple />
<button type="button" id="cancel-uploads">Cancel uploads</button>
<script src="main.js"></script>

// Assumes WorkerPool class is defined elsewhere (see below)
let pool = new WorkerPool('upload-worker.js')

const fileInput = document.querySelector('#file-input')

fileInput.addEventListener('change', (evt) => {
  if (pool.closed) pool = new WorkerPool('upload-worker.js')
  const files = Array.from(evt.target.files)
  files.forEach((file) => {
    console.log(`Queueing ${file.name} for upload...`)
    pool.processFile(file, {
      onProgress: (pct, msg) => updateProgressUI(file.name, pct, msg),
      onComplete: (msg) => showSuccess(file.name, msg),
      onError: (err) => showError(file.name, err),
    })
  })
  fileInput.value = ''
})

function updateProgressUI(filename, pct, message) {
  // Update your progress bar or UI element here
  console.log(`${filename}: ${pct.toFixed(1)}% – ${message}`)
}

function showSuccess(filename, message) {
  // Update UI to show completion
  console.info(`${filename}: ${message}`)
}

function showError(filename, error) {
  // Update UI to show error state
  console.error(`${filename}: Upload failed - ${error}`)
}

document.getElementById('cancel-uploads').addEventListener('click', () => pool.terminate())
window.addEventListener('pagehide', () => {
  pool.terminate()
})

Worker implementation (upload-worker.js)

This worker coalesces the browser's variable-sized stream chunks into 1 MiB requests. The backend must authenticate and authorize each uploadId, accept chunks idempotently by index, enforce limits, and finalize only after validating all chunks. /upload accepts the multipart fields below; /complete-upload accepts JSON with uploadId and totalChunks. Both must return a successful HTTP status. These custom endpoints are not supplied by this tutorial, and filenames must never be treated as unchecked storage paths. This example has request timeouts but no retry or reload recovery; use a maintained resumable protocol when you need those guarantees.

const CHUNK_SIZE = 1024 * 1024

async function* readChunks(file) {
  const reader = file.stream().getReader()
  let buffer = new Uint8Array(CHUNK_SIZE)
  let used = 0
  let finished = false
  try {
    while (true) {
      const { done, value } = await reader.read()
      if (done) {
        finished = true
        break
      }
      let offset = 0
      while (offset < value.length) {
        const length = Math.min(CHUNK_SIZE - used, value.length - offset)
        buffer.set(value.subarray(offset, offset + length), used)
        used += length
        offset += length
        if (used === CHUNK_SIZE) {
          yield buffer
          buffer = new Uint8Array(CHUNK_SIZE)
          used = 0
        }
      }
    }
    if (used > 0) yield buffer.subarray(0, used)
  } finally {
    if (!finished) await reader.cancel().catch(() => {})
    reader.releaseLock()
  }
}

async function uploadChunk(chunk, filename, index, uploadId, totalChunks) {
  const formData = new FormData()
  // Send chunk index for server-side reassembly
  formData.append('chunkIndex', index.toString())
  // Send the actual chunk data as a Blob
  formData.append('fileChunk', new Blob([chunk]), `${filename}.part${index}`)
  formData.append('uploadId', uploadId)
  formData.append('totalChunks', String(totalChunks))

  // Replace '/upload' with your actual back-end endpoint
  const res = await fetch('/upload', {
    method: 'POST',
    body: formData,
    signal: AbortSignal.timeout(60_000),
  })

  if (!res.ok) {
    throw new Error(`Chunk rejected: HTTP ${res.status}`)
  }

}

self.onmessage = async (event) => {
  const file = event.data
  try {
    if (!(file instanceof File) || file.size === 0) throw new Error('Select a nonempty file')
    const uploadId = crypto.randomUUID()
    const totalChunks = Math.ceil(file.size / CHUNK_SIZE)
    let index = 0
    let uploaded = 0
    for await (const chunk of readChunks(file)) {
      await uploadChunk(chunk, file.name, index++, uploadId, totalChunks)
      uploaded += chunk.length
      self.postMessage({ type: 'progress', progress: uploaded / file.size * 100, message: 'Uploading.' })
    }
    const response = await fetch('/complete-upload', {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({ uploadId, totalChunks }),
      signal: AbortSignal.timeout(60_000),
    })
    if (!response.ok) throw new Error('Finalization failed')
    self.postMessage({ type: 'complete', message: 'Upload finished successfully.' })
  } catch {
    self.postMessage({ type: 'error', message: 'Upload failed. Please try again.' })
  }
}

Why not read the whole file once in the worker?

While reading the entire file within the worker avoids blocking the main thread, it still consumes significant memory within the worker thread itself. Streaming the file chunk-by-chunk inside the worker offers several advantages:

• Lower Memory Footprint: Only small chunks reside in memory at any given time.
• Backpressure: The next application chunk is requested only after the current upload completes. Retry and resume support still require a server protocol and saved acknowledged state.
• Parallel Chunk Uploads: Advanced implementations could potentially upload multiple chunks concurrently (though this adds complexity in ordering and server-side handling).

Building a tiny worker pool

Using a single worker can still become a bottleneck if you need to process many files simultaneously. A WorkerPool limits concurrent tasks and queues pending files. Start with a small limit and measure; CPU count is not an upload-bandwidth recommendation. This pool cancels all work on a worker script or message-serialization failure, rather than reusing a broken worker.

class WorkerPool {
  constructor(script, size = Math.min(navigator.hardwareConcurrency || 2, 4)) {
    if (!Number.isInteger(size) || size < 1 || size > 4) throw new Error('Use 1 to 4 workers')
    this.closed = false
    this.workers = []
    this.idleWorkers = []
    this.taskQueue = []
    this.taskCallbacks = new Map() // Map task ID to callbacks

    console.log(`Initializing WorkerPool with size ${size}`)
    for (let i = 0; i < size; i++) {
      const worker = new Worker(script)
      worker.id = `worker_${i}`
      // Handle messages from the worker
      worker.onmessage = (e) => this.handleWorkerMessage(worker, e.data)
      // Handle errors occurring within the worker itself
      worker.onerror = (e) => this.handleWorkerError(worker, e)
      worker.onmessageerror = (e) => this.handleWorkerError(worker, e)
      this.workers.push(worker)
      this.idleWorkers.push(worker)
    }
  }

  generateTaskId() {
    return crypto.randomUUID()
  }

  processFile(file, callbacks) {
    if (this.closed) {
      callbacks.onError?.('The upload pool is closed.')
      return
    }
    const taskId = this.generateTaskId()
    const task = { id: taskId, file }
    this.taskCallbacks.set(taskId, callbacks)

    const idleWorker = this.idleWorkers.pop()
    if (idleWorker) {
      // An idle worker is available, run the task immediately
      this.runTask(idleWorker, task)
    } else {
      // All workers are busy, add task to the queue
      this.taskQueue.push(task)
      console.log(
        `Worker pool busy. Queued task ${taskId} for ${file.name}. Queue size: ${this.taskQueue.length}`,
      )
    }
  }

  runTask(worker, task) {
    console.log(`Assigning task ${task.id} (${task.file.name}) to ${worker.id}`)
    worker.currentTask = task // Associate task metadata with the worker
    // Send the file object to the worker to start processing
    // For large files, consider if Transferable Objects are applicable/needed
    try {
      worker.postMessage(task.file)
    } catch (error) {
      this.handleWorkerError(worker, error)
    }
  }

  handleWorkerMessage(worker, data) {
    const task = worker.currentTask
    if (!task) {
      console.warn(`Received message from worker ${worker.id} without an assigned task.`)
      return
    }

    const callbacks = this.taskCallbacks.get(task.id)
    if (!callbacks) {
      console.warn(`Received message for unknown or completed task ${task.id}`)
      return // Task might have been cancelled or already completed/failed
    }

    // Process messages based on their type
    switch (data.type) {
      case 'progress':
        if (callbacks.onProgress) callbacks.onProgress(data.progress, data.message)
        break
      case 'complete':
        this.finishTask(worker, task.id, () => callbacks.onComplete?.(data.message))
        break
      case 'error':
        this.finishTask(worker, task.id, () => callbacks.onError?.(data.message))
        break
      default:
        console.warn(`Received unknown message type from worker ${worker.id}:`, data.type)
    }
  }

  handleWorkerError(worker, errorEvent) {
    errorEvent.preventDefault?.()
    this.terminate('An upload worker failed. Select your files to try again.')
  }

  finishTask(worker, taskId, notify) {
    // Detach completed work before user callbacks can cancel or enqueue more work.
    worker.currentTask = null
    this.taskCallbacks.delete(taskId)
    try {
      notify()
    } finally {
      // A callback may terminate the pool; never return a dead worker to it.
      if (!this.closed) {
        const nextTask = this.taskQueue.shift()
        if (nextTask) this.runTask(worker, nextTask)
        else this.idleWorkers.push(worker)
      }
    }
  }

  terminate(message = 'Upload canceled.') {
    if (this.closed) return
    this.closed = true
    console.log('Terminating worker pool...')
    this.workers.forEach((worker) => {
      console.log(`Terminating worker ${worker.id}`)
      worker.terminate()
    })
    // Clear internal state
    this.workers = []
    this.idleWorkers = []
    this.taskQueue = []
    const callbacks = [...this.taskCallbacks.values()]
    this.taskCallbacks.clear()
    const errors = []
    for (const callback of callbacks) {
      try {
        callback.onError?.(message)
      } catch (error) {
        errors.push(error)
      }
    }
    if (errors.length > 0) throw new AggregateError(errors, 'Upload cancellation callbacks failed.')
  }
}

Browser compatibility

Web APIs evolve, so always verify browser support for the features you rely on.

The Blob.stream()⁠ method is inherited by File, so these are not separate compatibility requirements. Also check Worker⁠ and AbortSignal.timeout()⁠ in your supported browsers. This example sends a File by structured clone; it does not require transferable-stream support.

For browsers lacking support for File.stream(), you might need to fall back to a FileReader-based approach (potentially within the worker to avoid blocking the main thread, but still using more memory) or use established libraries like tus-js-client or Uppy, which handle compatibility and provide features like resumability.

Memory management best practices

• Limit Worker Count: Start with a small limit and measure CPU, memory and network behavior. Creating too many workers can lead to excessive context switching and memory overhead.
• Terminate Workers: Explicitly call worker.terminate() or pool.terminate() when the workers are no longer needed (e.g., after all uploads complete, or on page unload) to release resources. Use try...finally blocks in your application logic to ensure termination happens even if errors occur during the upload process.
• Release References: In both the main thread and workers, nullify references to large objects (like File objects, Blobs, ArrayBuffers, or stream readers) once they are no longer needed (reader = null, file = null, chunk = null) to allow garbage collection. Ensure stream readers are released using reader.releaseLock(). Cancel unfinished streams as well; releasing a lock does not close the stream or cancel its producer.
• Chunk Size: Choose a sensible chunk size (e.g., 1-10 MiB). Very small chunks increase network overhead (more HTTP requests per file), while very large chunks negate some of the memory-saving benefits of streaming.
• Monitor Memory: Use browser developer tools (like Chrome's Memory panel or Firefox's Memory tool) during development and testing to monitor memory usage under load and identify potential leaks.

The main-thread example terminates the pool on Cancel uploads or pagehide, then creates a fresh pool when the user selects files again. Do not terminate immediately after queueing asynchronous work unless you intend to cancel it. Server-side expiration must clean up bytes already received.

Security and resilience

1. CORS: Configure your upload endpoint's Cross-Origin Resource Sharing (CORS) policy carefully on the server. Allow only necessary HTTP methods (POST, potentially OPTIONS for preflight requests), the headers your chosen protocol actually uses, and restrict origins (Access-Control-Allow-Origin) to your application's domain.
2. Authentication/Authorization: Secure your upload endpoint. For chunked uploads, ensure each chunk request is authenticated and authorized. Methods include using secure HTTP-only session cookies, bearer tokens (JWTs) sent in the Authorization header, or generating pre-signed URLs for each chunk or the entire upload session (common with cloud storage).
3. Retries: Network issues are common. Implement a retry mechanism in your uploadChunk function for failed chunk uploads. Use exponential back-off (waiting progressively longer between retries: e.g., 1s, 2s, 4s) to avoid overwhelming the server or network. Abort retrying after a reasonable number of attempts (e.g., 3-5).
4. Cancellation: Provide users with a way to cancel ongoing uploads. Use the AbortController API. Create an AbortController instance before starting the upload, pass its signal to each fetch request, and call controller.abort() when the user cancels. Ensure your error handling catches the AbortError.

An AbortController belongs in the same worker as the request. A main-thread controller is not automatically shared with that worker. This example instead stops all active workers on Cancel uploads; per-file cancellation would need an explicit worker message and queue-removal contract.

Debugging Web Workers

Debugging workers can be slightly different from main thread debugging:

• Browser DevTools: In Chrome's Sources panel, select the worker in the Threads pane to change the debugging context⁠. In Firefox's Debugger, open an active worker's source file. Both browsers support breakpoints, variable inspection and console.log output for workers; see debugging worker threads⁠.
• Error Handling: Robust postMessage communication for errors (as shown in the examples) is crucial for understanding issues occurring within the worker, as direct try...catch from the main thread won't catch worker errors. Ensure worker errors are explicitly caught and posted back.

Common pitfalls

• Excessive Workers: Spawning a new worker for every file instead of using a pool can overwhelm the system's resources (CPU and memory).
• Stream Locks: Forgetting to call reader.releaseLock() on a ReadableStreamDefaultReader after finishing reading or encountering an error. Always use a finally block for releaseLock() and cancel an unfinished stream before releasing it.
• Large Message Payloads: Avoid posting very large data objects between the main thread and workers using postMessage, as this involves serialization and deserialization overhead (or structured cloning). For large binary data, investigate using Transferable objects (like ArrayBuffer) for more efficient zero-copy transfers where supported and appropriate.
• Chunk Ordering: Assuming the server will receive chunks in the exact order they were sent. Network latency and concurrent requests can cause reordering. Always include an index or byte offset with each chunk so the server can reassemble the file correctly.
• Unhandled Errors: Lack of proper try...catch blocks within the worker, especially around asynchronous operations like stream reading (reader.read()) and network requests (fetch), can cause silent failures or unhandled promise rejections within the worker.

Key takeaways

• Web Workers can move CPU-intensive file processing off the main thread, keeping the UI responsive during uploads.
• JavaScript streams allow efficient handling of large files by processing data in chunks, reducing application buffering. Resumability is a separate protocol concern.
• A worker pool bounds concurrent processing, helping control resource use when handling multiple simultaneous uploads.
• Robust error handling (including network retries and stream error handling), proper resource cleanup (terminate, releaseLock), security considerations (CORS, auth), and attention to browser compatibility are vital for production-ready implementations.

For a production-ready solution that handles chunking, resumability, retries, and parallel uploads out of the box, consider using libraries like Uppy with its various upload plugins, or explore services designed for robust file handling. Transloadit's handling uploads service integrates these concepts for reliable large file uploads. Happy uploading!