Last updated: May 12, 2025

<span aria-hidden="true" id="real-time-video-filters-in-browsers-with-ffmpeg-and-webcodecs"></span>

# Real-time video filters in browsers with FFmpeg and webcodecs

![Tim Koschützki](/assets/images/teammates/avatar-tim-kos-1.jpg?dpl=dpl_5zDg5SaYi7KaFpi2w3C4HPYbgapj)

#### Tim Koschützki

Co-founder · Berlin, Germany · Show bio

[](https://x.com/tim%5Fkos)[](https://github.com/tim-kos)

Real-time video processing directly in browsers is finally practical, thanks to WebAssembly and modern browser APIs. In this DevTip, you’ll learn how to combine FFmpeg compiled to WebAssembly with the WebCodecs API to apply dynamic video filters on the client side.

<span aria-hidden="true" id="introduction-to-real-time-video-processing-in-browsers"></span>

## Introduction to real-time video processing in browsers

Until recently, complex video manipulation demanded server-side infrastructure. Now, browsers can decode, transform, and re-encode frames locally using powerful APIs. Client-side processing lowers latency, keeps raw footage private by processing it on the user's device, and reduces back-end costs.

<span aria-hidden="true" id="overview-of-ffmpeg-and-webassembly"></span>

## Overview of FFmpeg and WebAssembly

[FFmpeg⁠](https://ffmpeg.org/) is a comprehensive, battle-tested multimedia toolkit that decodes, encodes, muxes, demuxes, streams, filters, and plays nearly any media format created. WebAssembly (Wasm) is a binary instruction format that allows code written in languages like C/C++ to run in the browser at near-native speed. By compiling FFmpeg to Wasm—projects such as[ffmpeg.wasm⁠](https://github.com/ffmpegwasm/ffmpeg.wasm) handle this complex task—you unlock FFmpeg’s powerful command-line capabilities directly within JavaScript.

<span aria-hidden="true" id="how-can-ffmpeg-be-used-directly-in-browsers"></span>

### How can FFmpeg be used directly in browsers?

1. Load the FFmpeg Wasm build into your web application.
2. Use the provided API to write input files (like video frames or existing videos) into FFmpeg’s virtual in-memory filesystem.
3. Execute FFmpeg commands using `ffmpeg.exec()` with familiar CLI arguments (e.g., `-vf 'hue=s=0'`for grayscale).
4. Read the processed output files back from the virtual filesystem.

<span aria-hidden="true" id="what-are-the-benefits-of-using-webassembly-for-video-processing"></span>

### What are the benefits of using WebAssembly for video processing?

WebAssembly provides near-native execution speed for computationally intensive tasks like video encoding and filtering, offers more consistent performance across different browsers compared to JavaScript implementations, and enables the use of mature, feature-rich C/C++ libraries like FFmpeg without requiring server round-trips for processing.

<span aria-hidden="true" id="setting-up-ffmpeg-in-the-browser-using-webassembly"></span>

## Setting up FFmpeg in the browser using WebAssembly

First, install the necessary packages:

```bash
npm install @ffmpeg/ffmpeg @ffmpeg/util

```

Then, load FFmpeg in your JavaScript code:

```javascript
import { FFmpeg } from '@ffmpeg/ffmpeg'
import { fetchFile } from '@ffmpeg/util' // Optional utility for fetching files

const ffmpeg = new FFmpeg()

async function loadFFmpeg() {
  try {
    // Load the FFmpeg core script
    await ffmpeg.load()
    console.log('FFmpeg core loaded successfully.')
  } catch (error) {
    console.error('Error loading FFmpeg:', error)
    // Handle the error appropriately, e.g., disable FFmpeg features
  }
}

loadFFmpeg()

```

<span aria-hidden="true" id="introduction-to-the-webcodecs-api"></span>

## Introduction to the webcodecs API

The [WebCodecs API⁠](https://developer.mozilla.org/en-US/docs/Web/API/WebCodecs%5FAPI) provides low-level access to the browser's built-in media codecs (encoders and decoders). Instead of treating a `MediaStream` as an opaque object piped into a `<video>` element, WebCodecs allows you to work with individual audio chunks (`AudioData`) and video frames (`VideoFrame`). You can decode compressed streams, access and manipulate raw frame data (pixels), and encode frames back into a compressed format or display them.

<span aria-hidden="true" id="how-does-the-webcodecs-api-enhance-real-time-video-processing"></span>

### How does the webcodecs API enhance real-time video processing?

* **Direct Frame Access**: It allows direct manipulation of raw video frame data, eliminating inefficient workarounds like drawing frames to a canvas, exporting as an image (PNG/JPEG), processing, and then drawing back.
* **Efficient Encoding/Decoding**: Leverages the browser's media pipeline, which is often hardware-accelerated, for efficient encoding and decoding.
* **Stream Integration**: Integrates seamlessly with the Streams API, enabling the creation of efficient processing pipelines with built-in back-pressure management.

<span aria-hidden="true" id="browser-support"></span>

## Browser support

WebCodecs is available in:

* Chrome 94+
* Edge 94+
* Opera 80+
* Firefox (Supported from version 118, but may require enabling the `media.wmf.enabled` or`media.ffvpx.enabled` flag in `about:config` depending on the OS and version)

Safari does not yet support the WebCodecs API as of mid-2024.

FFmpeg.wasm relies on WebAssembly and `SharedArrayBuffer` support, which are widely available in modern browsers.

<span aria-hidden="true" id="security-requirements"></span>

## Security requirements

Both WebCodecs and `SharedArrayBuffer` (required by FFmpeg.wasm for multithreading) require a secure context (HTTPS). To enable `SharedArrayBuffer`, your server must also send the following HTTP headers:

```http
Cross-Origin-Opener-Policy: same-origin
Cross-Origin-Embedder-Policy: require-corp

```

These headers ensure your page is sufficiently isolated, which is necessary for security reasons when using powerful features like `SharedArrayBuffer`. Most hosting platforms and frameworks provide ways to configure these headers (e.g., via `nginx.conf`, `netlify.toml`, Vercel's `vercel.json`, or Next.js middleware).

<span aria-hidden="true" id="implementing-real-time-video-filters-with-ffmpeg-and-webcodecs"></span>

## Implementing real-time video filters with FFmpeg and webcodecs

Below is a conceptual example demonstrating how to apply a grayscale filter to a webcam stream using FFmpeg.wasm and WebCodecs. **Note:** This specific approach involves converting each frame to a PNG, processing it with FFmpeg, and then converting it back. While functional, this round-trip is **not performant** for high-resolution or high-FPS real-time filtering. More efficient methods would involve direct pixel manipulation using Canvas API or WebGL, or potentially more advanced FFmpeg.wasm techniques if available for direct memory access.

```javascript
async function startFilteredStream(videoEl) {
  if (!ffmpeg.loaded) {
    console.error('FFmpeg is not loaded yet.')
    alert('FFmpeg is not ready. Please wait or check the console.')
    return
  }

  // 1. Get webcam stream
  const stream = await navigator.mediaDevices.getUserMedia({ video: true })
  const inputTrack = stream.getVideoTracks()[0]

  // 2. Set up WebCodecs processing pipeline
  const processor = new MediaStreamTrackProcessor({ track: inputTrack })
  const generator = new MediaStreamTrackGenerator({ kind: 'video' })

  const transformer = new TransformStream({
    async transform(frame, controller) {
      try {
        // Draw frame to an OffscreenCanvas for conversion
        const { displayWidth: w, displayHeight: h } = frame
        const canvas = new OffscreenCanvas(w, h)
        const ctx = canvas.getContext('2d')
        ctx.drawImage(frame, 0, 0, w, h)

        // Convert canvas to PNG Blob, then ArrayBuffer
        const blob = await canvas.convertToBlob({ type: 'image/png' })
        const arrayBuffer = await blob.arrayBuffer()

        // Write input PNG to FFmpeg's virtual filesystem
        const inputFilename = 'in.png'
        const outputFilename = 'out.png'
        await ffmpeg.writeFile(inputFilename, new Uint8Array(arrayBuffer))

        // Execute FFmpeg command (grayscale filter)
        // Note: This is the performance bottleneck
        await ffmpeg.exec(['-i', inputFilename, '-vf', 'hue=s=0', outputFilename])

        // Read the processed PNG file
        const outputData = await ffmpeg.readFile(outputFilename)

        // Clean up files in virtual filesystem
        await ffmpeg.deleteFile(inputFilename)
        await ffmpeg.deleteFile(outputFilename)

        // Create an ImageBitmap from the output PNG data
        const outputBlob = new Blob([outputData.buffer], { type: 'image/png' })
        const bitmap = await createImageBitmap(outputBlob)

        // Create a new VideoFrame with the processed bitmap
        const newFrame = new VideoFrame(bitmap, {
          timestamp: frame.timestamp,
          duration: frame.duration,
        })

        // Enqueue the new frame into the output stream
        controller.enqueue(newFrame)
      } catch (error) {
        console.error('Error processing frame:', error)
        // Optionally, enqueue the original frame or skip
        // controller.enqueue(frame);
      } finally {
        // IMPORTANT: Close the original frame to release resources
        frame.close()
      }
    },
  })

  // Pipe the processor through the transformer to the generator
  processor.readable
    .pipeThrough(transformer)
    .pipeTo(generator.writable)
    .catch((e) => console.error('Pipeline error:', e))

  // Display the output stream in the video element
  videoEl.srcObject = new MediaStream([generator])
  await videoEl.play()
}

// Example usage:
const videoElement = document.querySelector('video')
if (videoElement) {
  // Ensure FFmpeg is loaded before starting
  const checkLoadInterval = setInterval(() => {
    if (ffmpeg.loaded) {
      clearInterval(checkLoadInterval)
      startFilteredStream(videoElement).catch(console.error)
    }
  }, 100)
} else {
  console.error('No video element found on the page.')
}

```

<span aria-hidden="true" id="performance-considerations-and-optimization-tips"></span>

## Performance considerations and optimization tips

* **Avoid Blocking the Main Thread**: Heavy processing like the FFmpeg execution in the example should ideally run in a Web Worker to avoid freezing the UI. Transfer `VideoFrame` objects to/from the worker using `postMessage` (they are transferable objects).
* **Resource Management**: Always call `frame.close()` on `VideoFrame` objects when you are finished with them to release underlying memory and GPU resources promptly. Failure to do so can lead to memory leaks and performance degradation.
* **Minimize Data Copies**: The PNG conversion round-trip in the example is inefficient. For simpler filters (brightness, contrast, saturation, basic color transforms), use the Canvas 2D API or WebGL directly on the `VideoFrame` data for much better performance. `ctx.drawImage(frame, ...)` is efficient.
* **Control Frame Rate/Resolution**: If processing is too slow, consider requesting a lower resolution or frame rate from `getUserMedia`, or selectively dropping frames within the`TransformStream`.
* **Monitor Pipeline Health**: Check the `readable.locked` and `writable.locked` properties of streams, or monitor queue sizes (`transformer.readable.controller.desiredSize`) to understand back-pressure and potential bottlenecks.
* **Hardware Acceleration**: WebCodecs often uses hardware acceleration for encoding/decoding. Ensure your processing logic doesn't become the bottleneck that negates these benefits.

<span aria-hidden="true" id="conclusion-and-potential-use-cases"></span>

## Conclusion and potential use cases

Combining WebAssembly (via FFmpeg.wasm) and the WebCodecs API opens up exciting possibilities for client-side video manipulation directly in the browser. While real-time filtering using complex FFmpeg operations frame-by-frame faces performance challenges with current methods, the underlying technologies are powerful. Potential applications include:

* Applying simple filters (brightness, contrast) or overlays in video chat applications.
* Implementing virtual backgrounds (though often better suited to specialized ML models).
* Creating lightweight, client-side video editing tools for trimming or simple effects.
* Privacy-preserving blurring or masking of video content before upload.

What are practical use cases for real-time video filters in web applications? They range from enhancing user experience in communication apps to enabling novel creative tools and improving privacy, all while reducing server load.

While client-side processing is advancing rapidly, complex, high-quality video encoding and manipulation at scale often still benefit from robust server-side solutions. If you need reliable, scalable video processing in production, consider services like Transloadit's[video encoding platform](/services/video-encoding.md), powered by versatile Robots like[🤖 /video/encode](/docs/robots/video-encode.md).

Experiment with WebCodecs and FFmpeg.wasm to see what you can build directly in the browser!

\#ffmpeg#webassembly#webcodecs-api#real-time-video-processing#browser-video-filters#webcam-stream-processing#video-encoding-service

### 👩‍💻 Join 20k+ developers

Sign up for our [monthly newsletter](/newsletters.md) to receive direct links to 3 exclusive tech — and 2 product updates. No less, no more.

Your email:

Get access

## File uploading and encoding. Made simple.

Transloadit streamlines file handling for developers, trusted by brands like Coursera and The New York Times. We’re known for a reliable API, top-notch support, and a strong commitment to open source, with projects like [Uppy⁠](https://uppy.io) and [Tus⁠](https://tus.io) setting standards in file processing.

[Sign up](/c/)[Book a Demo](https://survey.typeform.com/to/kRg47Xi5)

No credit card needed · 5 GB included in the free plan

Cancel anytime