Experimental webcam filters with FFmpeg.wasm and WebCodecs

Tim Koschützki

Co-founder · Berlin, Germany ·

Build a local webcam preview that runs FFmpeg’s grayscale filter in your browser. You will see the camera preview and a filtered canvas, with controls to stop and restart processing. Each small frame makes a PNG round trip through FFmpeg.wasm, so expect a CPU-heavy experiment with a low frame rate.

Follow a frame through the filter

The path is: camera video → VideoFrame → PNG → FFmpeg’s in-memory filesystem → grayscale PNG → canvas. FFmpeg’s hue=s=0 filter removes saturation. The app waits for each job to finish before requesting another frame; it does not queue every camera frame for processing. Each result ends its FFmpeg worker, so the next frame reloads the core. Repeatedly calling exec() on the same pinned core can hit a WebAssembly memory error in this loop. Using one execution per worker avoids that failure, at the cost of extra startup work.

WebAssembly lets us reuse FFmpeg’s filters locally, but it does not make this pipeline as fast as native FFmpeg. The ffmpeg.wasm FAQ explicitly cautions about that performance gap. The PNG encoding, decoding, and copies add more work.

Here, WebCodecs supplies a VideoFrame snapshot of the video. We close it after drawing its pixels. We do not use VideoEncoder or VideoDecoder, and this example makes no hardware-acceleration claim. The output is a canvas preview; it does not produce a recording or a filtered MediaStream for a video call.

Browser support

The walkthrough was tested in Chromium 145.0.7632.6 on Linux with a synthetic camera. A physical webcam and other browsers were not tested. The code checks VideoFrame, OffscreenCanvas, and requestVideoFrameCallback() before opening the camera.

This version takes snapshots from a playing video rather than using MediaStreamTrackProcessor or MediaStreamTrackGenerator. Those insertable-stream APIs have incompatible window and worker exposure across browsers. A frame callback is enough for this serial preview.

Security requirements

Open the page through localhost. Camera access requires a secure context and the user’s permission; HTTPS is required when serving it from a regular remote origin. Do not open index.html directly.

We use the single-threaded @ffmpeg/core, served from the same local server as the page. It does not require SharedArrayBuffer or cross-origin isolation headers. The separate @ffmpeg/core-mt build needs shared memory, cross-origin isolation, and an additional worker asset; switching to it is outside this walkthrough. Single-threaded describes the FFmpeg core: the wrapper still runs it in a Web Worker.

Create the local project

You need a Bash-compatible terminal, Node.js 22.12 or newer, and Corepack available to run Yarn. The tested setup used Node.js 26.8.1, Yarn 4.12.0, and Vite 8.3.0. Vite provides the module and worker handling; its manual installation guide explains the HTML entry point.

Run this from a parent directory where webcam-filter does not exist. The && chain stops on a failed step, and mkdir refuses to overwrite an existing project. Continue only if the whole block succeeds. The core files are copied from the pinned package, so the browser does not fetch an engine from a third-party CDN.

mkdir webcam-filter &&
cd webcam-filter &&
printf '{"private":true,"type":"module","packageManager":"yarn@4.12.0"}\n' > package.json &&
printf 'nodeLinker: node-modules\n' > .yarnrc.yml &&
touch yarn.lock &&
corepack yarn add --exact @ffmpeg/ffmpeg@0.12.15 @ffmpeg/core@0.12.10 &&
corepack yarn add --dev --exact vite@8.3.0 &&
mkdir -p public/ffmpeg &&
cp node_modules/@ffmpeg/core/dist/esm/ffmpeg-core.{js,wasm} public/ffmpeg/

In webcam-filter, create vite.config.ts. Excluding the FFmpeg wrapper from dependency pre-bundling keeps its module worker URL attached to the package.

import { defineConfig } from 'vite'

export default defineConfig({
  optimizeDeps: { exclude: ['@ffmpeg/ffmpeg'] },
})

Create index.html in the same directory:

<!doctype html>
<html lang="en">
  <head>
    <meta charset="utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1" />
    <link rel="icon" href="data:," />
    <title>FFmpeg webcam experiment</title>
  </head>
  <body>
    <h1>FFmpeg webcam experiment</h1>
    <button id="start" type="button">Start camera</button>
    <button id="stop" type="button" disabled>Stop</button>
    <p id="status" role="status">Ready.</p>
    <figure>
      <figcaption>Camera</figcaption>
      <video id="camera" aria-label="Camera preview" width="320" height="240" muted playsinline></video>
    </figure>
    <figure>
      <figcaption>FFmpeg grayscale</figcaption>
      <canvas id="output" aria-label="Grayscale preview" width="320" height="240"></canvas>
    </figure>
    <script type="module" src="/main.ts"></script>
  </body>
</html>

Process one frame at a time

Create main.ts. Each start owns an FFmpeg wrapper and camera stream. Stop clears the current session immediately; later promises must still belong to that session before they can update the preview. This matters when permission, engine loading, or PNG decoding finishes after Stop.

import { FFmpeg } from '@ffmpeg/ffmpeg'

const startButton = document.querySelector('#start')
const stopButton = document.querySelector('#stop')
const status = document.querySelector('#status')
const camera = document.querySelector('#camera')
const output = document.querySelector('#output')
if (
  !(startButton instanceof HTMLButtonElement) ||
  !(stopButton instanceof HTMLButtonElement) ||
  !(status instanceof HTMLElement) ||
  !(camera instanceof HTMLVideoElement) ||
  !(output instanceof HTMLCanvasElement)
) {
  throw new Error('Missing demo elements')
}
const outputContext = output.getContext('2d')
if (!outputContext) throw new Error('Canvas 2D is unavailable')

interface Session {
  ffmpeg: FFmpeg
  canvas: OffscreenCanvas
  stream?: MediaStream
  callbackId?: number
}
let current: Session | undefined

const loadFFmpeg = async (session: Session): Promise<void> => {
  if (session.ffmpeg.loaded) return
  await session.ffmpeg.load({
    coreURL: new URL('/ffmpeg/ffmpeg-core.js', location.href).href,
    wasmURL: new URL('/ffmpeg/ffmpeg-core.wasm', location.href).href,
  })
}

const stop = (message = 'Stopped.'): void => {
  const session = current
  if (!session) return
  current = undefined
  if (session.callbackId !== undefined) {
    camera.cancelVideoFrameCallback(session.callbackId)
  }
  session.ffmpeg.terminate()
  for (const track of session.stream?.getTracks() ?? []) track.stop()
  camera.pause()
  camera.srcObject = null
  outputContext.clearRect(0, 0, output.width, output.height)
  status.textContent = message
  startButton.disabled = false
  stopButton.disabled = true
}

const schedule = (session: Session): void => {
  if (current !== session) return
  session.callbackId = camera.requestVideoFrameCallback((_now, metadata) => {
    session.callbackId = undefined
    void filterFrame(session, metadata.mediaTime)
  })
}

const filterFrame = async (session: Session, mediaTime: number): Promise<void> => {
  try {
    if (current !== session) return
    const context = session.canvas.getContext('2d')
    if (!context) throw new Error('Canvas 2D is unavailable')
    const frame = new VideoFrame(camera, { timestamp: Math.round(mediaTime * 1_000_000) })
    try {
      context.drawImage(frame, 0, 0, 320, 240)
    } finally {
      frame.close()
    }
    const blob = await session.canvas.convertToBlob({ type: 'image/png' })
    const bytes = new Uint8Array(await blob.arrayBuffer())
    if (current !== session) return
    await loadFFmpeg(session)
    if (current !== session) return
    await session.ffmpeg.writeFile('in.png', bytes)
    const code = await session.ffmpeg.exec([
      '-y', '-i', 'in.png', '-vf', 'hue=s=0', '-frames:v', '1', '-update', '1', 'out.png',
    ])
    if (code !== 0) throw new Error('FFmpeg failed')
    const data = await session.ffmpeg.readFile('out.png')
    if (!(data instanceof Uint8Array) || data.length === 0) {
      throw new Error('FFmpeg produced no image')
    }
    const bitmap = await createImageBitmap(
      new Blob([new Uint8Array(data)], { type: 'image/png' }),
    )
    try {
      if (current !== session) return
      outputContext.drawImage(bitmap, 0, 0)
      status.textContent = 'Filtering…'
    } finally {
      bitmap.close()
    }
    // This per-frame experiment uses one exec() per worker.
    session.ffmpeg.terminate()
    schedule(session)
  } catch {
    if (current === session) stop('Filter failed. Start again to retry.')
  }
}

startButton.onclick = async () => {
  if (current) return
  if (
    !navigator.mediaDevices?.getUserMedia ||
    typeof VideoFrame !== 'function' ||
    typeof OffscreenCanvas !== 'function' ||
    typeof createImageBitmap !== 'function' ||
    typeof camera.requestVideoFrameCallback !== 'function' ||
    typeof Worker !== 'function' ||
    typeof WebAssembly !== 'object'
  ) {
    status.textContent = 'Required browser APIs are unavailable. Use a supported browser on localhost.'
    return
  }
  const session: Session = { ffmpeg: new FFmpeg(), canvas: new OffscreenCanvas(320, 240) }
  current = session
  startButton.disabled = true
  stopButton.disabled = false
  status.textContent = 'Loading FFmpeg…'
  let failureMessage = 'Could not load FFmpeg. Check the local server and engine files, then retry.'
  try {
    await loadFFmpeg(session)
    if (current !== session) return
    status.textContent = 'Requesting camera…'
    failureMessage = 'Could not open camera. Check camera permission and availability, then retry.'
    const stream = await navigator.mediaDevices.getUserMedia({
      audio: false,
      video: { width: { ideal: 320 }, height: { ideal: 240 }, frameRate: { ideal: 5 } },
    })
    // getUserMedia has no abort signal; release a stream granted after Stop.
    if (current !== session) {
      for (const track of stream.getTracks()) track.stop()
      return
    }
    session.stream = stream
    const track = stream.getVideoTracks()[0]
    if (!track) throw new Error('No video track')
    track.addEventListener('ended', () => {
      if (current === session) stop('Camera ended. Start again to retry.')
    }, { once: true })
    camera.srcObject = stream
    await camera.play()
    if (current !== session) return
    status.textContent = 'Waiting for a frame…'
    schedule(session)
  } catch {
    if (current === session) stop(failureMessage)
  }
}
stopButton.onclick = () => stop()
window.addEventListener('pagehide', () => stop())

The camera dimensions and frame rate are preferences, not guarantees. Drawing into the fixed canvas scales every input to 320 × 240, even if the camera provides something larger or a different aspect ratio. The -update 1 option writes a single output image, and -y permits replacing that temporary filename. Terminating the worker after each frame, or on Stop or failure, releases its in-memory filesystem. The app saves no images to disk. VideoFrame.close() and ImageBitmap.close() release the image resources we create.

Run and stop the preview

From webcam-filter, start the server in the foreground:

corepack yarn vite --host 127.0.0.1

Open the local URL printed by Vite. Click Start camera and grant camera access. After Loading FFmpeg… and Requesting camera…, the status becomes Filtering… when the first grayscale frame appears. Hold something colorful in view to compare the two previews. Frames stay in the browser; this app neither uploads nor records them.

Click Stop to stop this app’s camera tracks, terminate its FFmpeg worker, and clear the previews. Start camera becomes available again, including when you stop during initialization. Browsers do not provide an API to dismiss a pending camera permission prompt. If you grant it after stopping, the late stream is stopped without updating the page. Navigating away also runs cleanup. Stop the terminal server with Ctrl+C when finished.

If permission is denied, the visible error asks you to check camera permission and availability. Allow access in the browser’s site settings and click Start camera again. A missing engine file produces a load error before the camera is requested; check that both files exist under public/ffmpeg. A processing error stops the session and shows Filter failed. Start again to retry.. If the camera ends, the message is Camera ended. Start again to retry..

Decide whether FFmpeg belongs in your preview

This loop skips camera frames while FFmpeg is busy. Lowering the requested frame rate can reduce work, but requesting five frames per second does not promise five filtered frames per second. Measure the whole PNG round trip on your target hardware before increasing the dimensions or adding more filters. A multithreaded core does not remove the image conversions and data copies.

For a grayscale camera preview, applying a Canvas 2D filter or a WebGL shader avoids that PNG and filesystem round trip. Keep this FFmpeg.wasm version for learning the file-based API or trying an FFmpeg filter on small snapshots. Recording, audio, and a filtered stream for WebRTC require a different output path.