Optimizing online file uploads with chunking and parallel uploads

Kevin van Zonneveld

Co-founder · Amsterdam, The Netherlands ·

Handling file uploads efficiently is a critical aspect of modern web applications. Large files can lead to slow upload times, network interruptions, and poor user experience. In this post, we explore advanced techniques—such as chunking and parallel uploading—for optimizing file uploads, ensuring faster and more reliable performance.

Introduction to file upload challenges

Uploading large files over the internet poses several challenges. Users may experience slow upload speeds due to bandwidth limitations or network instability, and interruptions often force restarts that lead to frustration. Modern web applications require robust upload systems that can handle these challenges while maintaining a seamless user experience.

Explaining chunking and why it matters

Chunking involves splitting a large file into smaller pieces, or chunks. This approach offers several advantages:

Retrying individual chunks instead of restarting the whole file
Better browser memory management
Easier progress tracking
Reduced impact from network interruptions
More efficient error recovery

The optimal chunk size depends on the protocol, network and server limits. This example targets at most ten chunks for small files and caps each chunk at 5 MiB as an application choice, not a browser limit. Empty files are rejected explicitly:

const calculateChunkSize = (fileSize) => {
  if (!Number.isSafeInteger(fileSize) || fileSize <= 0) {
    throw new Error('Select a nonempty file')
  }
  const MAXIMUM_CHUNK_SIZE = 1024 * 1024 * 5
  return Math.min(MAXIMUM_CHUNK_SIZE, Math.ceil(fileSize / 10))
}

Setting up a basic file upload with JavaScript

Place the HTML first, then combine the JavaScript definitions below in one module. The example requires server endpoints implementing the custom protocol described in the next section; it is not a drop-in client for an arbitrary file-upload endpoint.

<div id="upload-container">
  <label for="file-input">Files to upload</label>
  <input type="file" id="file-input" multiple />
  <button id="upload-btn">Upload</button>
  <button id="cancel-btn">Cancel</button>
  <p id="progress" role="status"></p>
</div>

class FileUploader {
  constructor() {
    this.abortController = null
    this.setupEventListeners()
  }

  setupEventListeners() {
    const uploadBtn = document.getElementById('upload-btn')
    const cancelBtn = document.getElementById('cancel-btn')

    uploadBtn.addEventListener('click', () => this.handleUpload())
    cancelBtn.addEventListener('click', () => this.cancelUpload())
  }

  async handleUpload() {
    if (this.abortController) return
    const fileInput = document.getElementById('file-input')
    const files = fileInput.files

    if (files.length === 0) {
      document.getElementById('progress').textContent = 'Please select a file.'
      return
    }

    this.abortController = new AbortController()

    try {
      for (const file of files) {
        await this.uploadFile(file)
      }
      document.getElementById('progress').textContent = 'Upload complete.'
    } catch (error) {
      document.getElementById('progress').textContent =
        error.name === 'AbortError' ? 'Upload canceled.' : 'Upload failed. Please try again.'
    } finally {
      this.abortController = null
    }
  }

  async uploadFile(file) {
    const upload = new SecureUploader(file)
    await upload.upload(this.abortController.signal, (percent) => {
      this.updateProgress(file, percent)
    })
  }

  cancelUpload() {
    if (this.abortController) {
      this.abortController.abort()
    }
  }

  updateProgress(file, percentage) {
    const progress = document.getElementById('progress')
    progress.textContent = `${file.name}: ${Math.round(percentage)}%`
  }
}

const uploader = new FileUploader()

Implementing chunked uploads

The server must authenticate each request, authorize the uploadId for that user, enforce size and chunk-count limits, and store chunks idempotently by (uploadId, chunkNumber). Finalization must verify all chunks and their order before publishing the file. Do not use fileName as a filesystem path. Expire incomplete uploads on the server. This teaching protocol retries within a page session; it does not implement reload recovery or a complete backend.

Each request below expects a successful HTTP status; no JSON response body is required.

class ChunkedUploader {
  constructor(file, options = {}) {
    this.file = file
    this.uploadId = crypto.randomUUID()
    this.chunkSize = calculateChunkSize(file.size)
    this.totalChunks = Math.ceil(file.size / this.chunkSize)
    this.retryLimit = options.retryLimit ?? 3
    this.retryDelay = options.retryDelay ?? 1000
    this.concurrency = options.concurrency ?? 3
    if (![this.retryLimit, this.concurrency].every((value) => Number.isInteger(value) && value > 0)
        || !Number.isFinite(this.retryDelay) || this.retryDelay < 0) {
      throw new Error('Invalid upload options')
    }
  }

  async uploadChunk(chunk, chunkNumber, signal) {
    const formData = new FormData()
    formData.append('chunk', chunk)
    formData.append('fileName', this.file.name)
    formData.append('uploadId', this.uploadId)
    formData.append('chunkNumber', chunkNumber)
    formData.append('totalChunks', this.totalChunks)

    let attempts = 0
    while (attempts < this.retryLimit) {
      signal.throwIfAborted()
      try {
        const response = await fetch('/upload-chunk', {
          method: 'POST',
          body: formData,
          signal,
        })

        if (!response.ok) {
          throw new Error(`HTTP error! status: ${response.status}`)
        }

        return
      } catch (error) {
        signal.throwIfAborted()
        attempts++
        if (attempts === this.retryLimit) throw error
        await this.waitForRetry(this.retryDelay * 2 ** (attempts - 1), signal)
      }
    }
  }

  waitForRetry(milliseconds, signal) {
    signal.throwIfAborted()
    return new Promise((resolve, reject) => {
      const onAbort = () => {
        clearTimeout(timer)
        reject(signal.reason)
      }
      const timer = setTimeout(() => {
        signal.removeEventListener('abort', onAbort)
        resolve()
      }, milliseconds)
      signal.addEventListener('abort', onAbort, { once: true })
    })
  }

  async upload(signal, onProgress) {
    let uploadedBytes = 0

    // Upload chunks with a concurrency limit
    for (let i = 0; i < this.totalChunks; i += this.concurrency) {
      signal.throwIfAborted()
      const requests = []
      for (let number = i; number < Math.min(i + this.concurrency, this.totalChunks); number++) {
        const chunk = this.file.slice(number * this.chunkSize, (number + 1) * this.chunkSize)
        requests.push(this.uploadChunk(chunk, number, signal).then(() => {
          uploadedBytes += chunk.size
          onProgress?.(uploadedBytes / this.file.size * 100)
        }))
      }
      // Settle in-flight requests before reporting failure or allowing another upload.
      const results = await Promise.allSettled(requests)
      const failure = results.find((result) => result.status === 'rejected')
      if (failure) throw failure.reason
    }

    // Notify server that all chunks are uploaded
    const response = await fetch('/complete-upload', {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({
        fileName: this.file.name,
        uploadId: this.uploadId,
        totalChunks: this.totalChunks,
      }),
      signal,
    })
    if (!response.ok) throw new Error(`Finalization failed: HTTP ${response.status}`)
  }
}

Parallel uploading: enhancing speed

The approach above leverages parallel uploading by processing multiple chunks concurrently. This technique can improve throughput where one request does not saturate the connection. Measure it against your server and network: parallel requests also add overhead and may hit rate limits. The progress callback counts acknowledged bytes, not bytes currently in flight.

Handling errors and retries

Our implementation includes robust error handling through automatic retries, exponential backoff, and graceful cancellation using AbortController. This strategy ensures that transient network issues or server errors can be retried a bounded number of times. It is advisable to provide clear error messages and differentiate between network failures and application errors, allowing users to retry uploads when necessary.

Ensuring security during uploads

The following subclass runs basic client-side checks before sending any chunks. These are early feedback only: a matching header does not prove a file is safe, and clients can bypass this code. The server must independently validate content and size, authorize access, and apply malware scanning or Content Disarm & Reconstruction where appropriate.

class SecureUploader extends ChunkedUploader {
  async upload(signal, onProgress) {
    signal.throwIfAborted()
    await this.validateFile()
    return super.upload(signal, onProgress)
  }

  async validateFile() {
    // Validate file signature using allowed types
    const allowedTypes = ['image/jpeg', 'image/png', 'application/pdf']
    if (!allowedTypes.includes(this.file.type)) {
      throw new Error('Unsupported file type')
    }
    const header = new Uint8Array(await this.file.slice(0, 4).arrayBuffer())
    if (!this.validateFileSignature(header)) {
      throw new Error('Invalid file signature')
    }

    // Size validation
    const maxSize = 100 * 1024 * 1024 // 100 MiB
    if (this.file.size > maxSize) {
      throw new Error('File too large')
    }

  }

  validateFileSignature(header) {
    const signatures = {
      'image/jpeg': [0xff, 0xd8, 0xff],
      'image/png': [0x89, 0x50, 0x4e, 0x47],
      'application/pdf': [0x25, 0x50, 0x44, 0x46],
    }

    const signature = signatures[this.file.type]
    return signature !== undefined && signature.every((byte, i) => header[i] === byte)
  }
}

Additional security measures include:

Implementing Content Security Policy (CSP) headers.
Utilizing Content Disarm & Reconstruction (CDR) and antivirus scanning.
Enforcing strict validation of file type and size.

Best practices and optimization tips

Use Web Workers for intensive file processing tasks.
Implement client-side file compression where appropriate.
For reload recovery, persist a server-issued upload URL and reconcile its acknowledged offset; saving a progress percentage alone is not sufficient.
Monitor memory usage during large uploads.
Provide clear visual feedback on upload status.
Ensure proper cleanup of failed uploads.
Leverage modern browser features, such as Service Workers for background uploads and ReadableStream for efficient data handling. Service Worker lifetime is limited; it does not guarantee an upload will continue after the browser closes.

Conclusion: building efficient upload systems

Building a robust file upload system requires careful attention to performance, security, and user experience. The techniques discussed provide a sound foundation for implementing reliable file uploads in modern web applications. By leveraging chunking, parallel uploads, and modern browser APIs, you can build efficient and resilient upload systems. For a production-ready solution that provides a maintained upload UI, consider Uppy. The following is an alternative to the custom uploader above, not an additional chunking layer. XHRUpload sends whole files; use Uppy's tus plugin with a compatible tus server when you need resumable uploads. Install @uppy/core, @uppy/dashboard and @uppy/xhr-upload in your bundled application, and provide an authorized /upload endpoint:

import { Uppy } from '@uppy/core'
import Dashboard from '@uppy/dashboard'
import XHRUpload from '@uppy/xhr-upload'
import '@uppy/core/css/style.min.css'
import '@uppy/dashboard/css/style.min.css'

const uppy = new Uppy()
  .use(Dashboard, {
    inline: true,
    target: '#upload-container',
  })
  .use(XHRUpload, {
    endpoint: '/upload',
    formData: true,
    fieldName: 'file',
  })

These modern approaches, along with careful error handling and security validations, will help you create a seamless file upload experience.