Create ASCII art from videos with Lua and FFmpeg

Tim Koschützki

Co-founder · Berlin, Germany · Show bio

Use Lua to map each video frame’s brightness to text, then render those characters into a silent ASCII art video. The example below produces an 80-column text grid and a 640 × 360 MP4 at 10 frames per second, using FFmpeg for decoding and ImageMagick for drawing the text.

Choose a short input video

Start with a short, intact SDR video with square pixels. Faces, silhouettes, and large contrasting shapes survive the conversion better than small lettering or busy backgrounds. This is a sampled visual effect: it drops audio, color, and detail, and it does not preserve every source frame.

The workflow runs locally on Linux in Bash. It saves PNG frames, ASCII text, rendered PNGs, and the final video, so use a short clip while choosing the look. It processes frames sequentially and stops at the first detected failure.

Check the tools and create a working directory

You need Bash, Lua, FFmpeg and ffprobe, ImageMagick 7’s magick command, and a readable monospaced font file. LuaRocks and Lua image libraries are unnecessary. The commands here were tested with Lua 5.5.1, FFmpeg/ffprobe 9.0.1, ImageMagick 7.1.2-31, and Liberation Mono; the Lua program was also replayed with Lua 5.4.8 and the workflow with FFmpeg/ffprobe 6.1.1. These are tested versions, not a claim that every intervening release or another operating system has been tested. For a new Lua installation, use the current release: Lua 5.4 has reached its final release⁠.

Install these tools through your Linux distribution or their official downloads. Your FFmpeg build must include the libx264 encoder, and ImageMagick must support PNG and text rendering. Check the commands before proceeding:

lua -v && ffmpeg -version && ffprobe -version && magick -version

Create a new directory and enter it only if creation succeeds. If ascii-demo already exists, choose a different name; do not delete an existing project to follow the tutorial.

mkdir -- ascii-demo && cd -- ascii-demo

Copy your clip into this directory as input.mp4. If you want a reproducible starting point, this command creates a 1.3-second test pattern instead. It refuses to replace an existing input.mp4:

ffmpeg -nostdin -v error -n -f lavfi -i "testsrc2=size=320x180:rate=10:duration=1.3" \
  -c:v libx264 -threads 1 -pix_fmt yuv420p input.mp4

Extract video frames with FFmpeg

For a single-frame preview, extract a fresh sequence. The parentheses keep the shell options local to this block, and mkdir refuses to reuse an existing frames directory.

(
  set -e
  mkdir -- frames
  ffmpeg -nostdin -v error -xerror -n -i ./input.mp4 -map 0:v:0 \
    -vf "fps=10,scale=640:-2,setsar=1" -threads 1 -filter_threads 1 \
    -start_number 1 frames/frame_%08d.png
)

The FFmpeg fps filter⁠ samples at 10 FPS. scale=640:-2 keeps the input proportions while choosing an even height, and setsar=1 marks the resulting pixels as square. A 16:9 clip produces 640 × 360 PNGs named frame_00000001.png, frame_00000002.png, and so on. The supplied test clip produces 13 frames. Sampling rounds timestamps to a frame grid; arbitrary input durations can differ from the result by roughly one 0.1-second frame interval.

Convert one PNG to ASCII with Lua

Save this complete program as ascii.lua in ascii-demo. It probes the PNG, asks FFmpeg for one scaled RGB frame, and checks that every expected pixel arrived before constructing the text.

-- ascii.lua
local chars = " .:-=+*#@" -- Sparse to dense: white glyphs on a black background.
local columns = 80
local cellRatio = 0.5 -- Approximate character width divided by line height.

local function shellQuote(value)
  return "'" .. value:gsub("'", "'\\''") .. "'"
end

local function readCommand(command)
  local pipe = assert(io.popen(command, "r"))
  local data, readError = pipe:read("*a")
  local ok, reason, status = pipe:close()
  assert(data, readError)
  assert(ok, string.format("Subprocess failed (%s %s)", reason, status))
  return data
end

local function main()
  assert(arg[1] and not arg[3], "Usage: lua ascii.lua <PNG> [output.txt]")
  local input = arg[1]
  if input:sub(1, 1) ~= "/" then input = "./" .. input end
  assert(input:match("%.png$"), "Input must be a local .png file")
  assert(arg[2] ~= arg[1], "Input and output must be different files")
  local quoted = shellQuote(input)
  local dimensions = readCommand(
    "ffprobe -v error -f image2 -pattern_type none -select_streams v:0 " ..
    "-show_entries stream=width,height -of csv=s=x:p=0 -i " .. quoted
  )
  local width, height = dimensions:match("^(%d+)x(%d+)%s*$")
  width, height = tonumber(width), tonumber(height)
  assert(width and height and width > 0 and height > 0, "Invalid PNG dimensions")
  local rows = math.max(1, math.floor(height / width * columns * cellRatio + 0.5))
  local pixels = readCommand(string.format(
    "ffmpeg -nostdin -v error -xerror -err_detect explode -threads 1 " ..
    "-f image2 -pattern_type none -i %s -map 0:v:0 " ..
    "-vf scale=%d:%d -filter_threads 1 -frames:v 1 " ..
    "-threads 1 -f rawvideo -pix_fmt rgb24 -", quoted, columns, rows
  ))
  assert(#pixels == columns * rows * 3, "Incomplete or unexpected RGB pixel data")

  local lines = {}
  local position = 1
  for y = 1, rows do
    local line = {}
    for x = 1, columns do
      local r, g, b = pixels:byte(position, position + 2)
      local brightness = 0.299 * r + 0.587 * g + 0.114 * b
      local index = math.floor(brightness / 255 * (#chars - 1) + 0.5) + 1
      line[x] = chars:sub(index, index)
      position = position + 3
    end
    lines[y] = table.concat(line)
  end
  local text = table.concat(lines, "\n") .. "\n"
  if arg[2] then
    local output = assert(io.open(arg[2], "wb"))
    local written, writeError = output:write(text)
    local closed, closeError = output:close()
    assert(written, writeError)
    assert(closed, closeError)
  else
    assert(io.write(text))
    assert(io.flush())
  end
end

local ok, message = pcall(main)
if not ok then
  io.stderr:write("Error: " .. tostring(message) .. "\n")
  os.exit(1)
end

Closing the pipe matters: Lua’s file:close()⁠ returns the subprocess status for a handle opened with io.popen. Reading some bytes alone does not prove that FFmpeg succeeded. The program also checks both the output write and close, so a missing directory or full disk cannot be reported as a successful save.

Preview the first frame on a dark terminal background:

lua ascii.lua frames/frame_00000001.png

To save it, create the parent directory first. This command refuses to reuse ascii_frames; the Lua program itself replaces an existing text destination once conversion succeeds and writing begins. A failed write can leave a partial text file, so use its exit status before consuming it.

mkdir -- ascii_frames && lua ascii.lua frames/frame_00000001.png ascii_frames/frame_00000001.txt

For a 16:9 frame, expect 23 rows of 80 characters. Black pixels map to spaces and white pixels to @, with intermediate brightness mapped to the characters between them. The row calculation compensates approximately for characters being taller than they are wide; it is not a font metric.

Create an ASCII art video

Save the following as create_ascii_video.sh, next to ascii.lua. It extracts its own fresh frames from the original video, so it does not consume the preview directories. Give it an input path and a new output directory. It refuses an existing destination, retains intermediate files for inspection after failure, and publishes ascii_video.mp4 only after encoding, counting, and decoding the resulting video.

Use a local output directory whose path contains no % character. FFmpeg interprets % in image sequence paths, even inside shell quotes. Spaces and leading hyphens are supported. Run one instance at a time, and do not change its input or intermediate files while it works.

#!/usr/bin/env bash
set -euo pipefail
export LC_ALL=C

die() { printf 'Error: %s\n' "$*" >&2; exit 1; }
[[ $# -eq 2 ]] || die 'Usage: bash create_ascii_video.sh INPUT_VIDEO NEW_OUTPUT_DIRECTORY'
input=$1
run=$2
[[ $input = /* ]] || input="./$input"
[[ $run = /* ]] || run="./$run"
[[ $run != *%* ]] || die 'Output path must not contain %'
[[ -r $input && -f $input ]] || die 'Input video is not a readable file'
[[ -f ascii.lua ]] || die 'Run from the directory containing ascii.lua'
[[ ! -e $run && ! -L $run ]] || die 'Output directory already exists; choose a new name'
font=${FONT:-/usr/share/fonts/liberation/LiberationMono-Regular.ttf}
[[ -r $font && -f $font ]] || die 'Set FONT to a readable monospaced font file'
for tool in lua ffmpeg ffprobe magick; do
  command -v "$tool" >/dev/null || die "Missing tool: $tool"
done

mkdir -- "$run"
mkdir -- "$run/frames" "$run/ascii_frames" "$run/ascii_images"
temporary="$run/ascii_video.part.mp4"
trap 'rm -f -- "$temporary"' EXIT

# A reported FFmpeg error is a failure even if that build returns zero.
run_ffmpeg() {
  if ! ffmpeg -nostdin -v error -xerror "$@" 2>"$run/ffmpeg.log"; then
    cat -- "$run/ffmpeg.log" >&2
    die 'FFmpeg failed'
  fi
  [[ ! -s $run/ffmpeg.log ]] || { cat -- "$run/ffmpeg.log" >&2; die 'FFmpeg reported an error'; }
}

run_ffmpeg -n -threads 1 -i "$input" -map 0:v:0 \
  -vf 'fps=10,scale=640:-2,setsar=1' -threads 1 -filter_threads 1 \
  -start_number 1 "$run/frames/frame_%08d.png"
shopt -s nullglob
frames=("$run"/frames/frame_*.png)
[[ ${#frames[@]} -gt 0 ]] || die 'No video frames were extracted'

count=0
for frame in "${frames[@]}"; do
  count=$((count + 1))
  printf -v name 'frame_%08d' "$count"
  [[ $frame = "$run/frames/$name.png" ]] || die 'Frame sequence has a gap'
  text="$run/ascii_frames/$name.txt"
  image="$run/ascii_images/$name.png"
  lua ascii.lua "$frame" "$text"
  # Strip only the display copy’s final newline to avoid an extra blank label row.
  printf '%s' "$(< "$text")" | magick -background black -fill white \
    -font "$font" -pointsize 12 label:@- -resize 640x360 \
    -gravity center -extent 640x360 "$image"
  printf 'Rendered frame %d/%d\n' "$count" "${#frames[@]}"
done

run_ffmpeg -n -framerate 10 -start_number 1 -start_number_range 1 \
  -i "$run/ascii_images/frame_%08d.png" -c:v libx264 -threads 1 \
  -pix_fmt yuv420p -crf 18 "$temporary"
decoded=$(ffprobe -v error -select_streams v:0 -count_frames \
  -show_entries stream=nb_read_frames -of csv=p=0 "$temporary")
[[ $decoded = "$count" ]] || die 'Encoded frame count does not match extracted frames'
run_ffmpeg -threads 1 -i "$temporary" -map 0:v:0 -f null -
mv -- "$temporary" "$run/ascii_video.mp4"
printf 'Created %s with %d frames at 10 FPS\n' "$run/ascii_video.mp4" "$count"

Set FONT to the absolute path of an installed monospaced font. The default above is the tested Linux path for Liberation Mono; other distributions may place it elsewhere. You can find installed fonts with your system’s font tools. A proportional font will misalign the grid.

FONT=/usr/share/fonts/liberation/LiberationMono-Regular.ttf \
  bash create_ascii_video.sh input.mp4 ascii-run

Open ascii-run/ascii_video.mp4 in your video player. For the supplied test pattern, the output has 13 frames, runs for 1.3 seconds at 10 FPS, and contains one H.264 video stream with no audio. The first and last rendered frames are ascii-run/ascii_images/frame_00000001.png and ascii-run/ascii_images/frame_00000013.png.

ImageMagick’s label: renderer⁠ draws the preformatted grid without word wrapping. Resizing the label before padding keeps portrait grids on the canvas too. If ImageMagick denies reading @-, your installation’s security policy blocks text indirection; use a suitably configured local installation rather than disabling its global policy blindly.

Tune the appearance

Change columns in ascii.lua to capture more or less detail. More columns mean smaller glyphs when fitted into the fixed video canvas. Adjust cellRatio to the chosen font’s character width divided by its line height if the subject looks stretched. For a white background and black text, reverse chars and change the renderer’s background and fill together.

For weak contrast, try adding eq=contrast=1.3 after setsar=1 in the extraction filter, then run the script into a new directory. That is a visual adjustment, not a guarantee of improved quality. Keep the same extraction and playback rate: changing only the final -framerate changes the speed.

Diagnose a failed conversion

An existing output directory is a deliberate refusal, including after a failed run. Inspect its intermediate files and ffmpeg.log, then retry with a new directory name. This prevents old text or images from extending a shorter new clip. A Lua failure stops the loop before later valid frames can conceal it, and an encoding or final decode failure leaves no published ascii_video.mp4.

The checks detect failed subprocesses, short RGB data, failed text writes, and reported errors in the batch’s FFmpeg stages. They do not prove that the source video was intact: a decoder can recover damaged media without reporting an error. Inspect recognizable content near the end of your input and output, as well as the extracted frame count. For the sampled sequence, the final video duration is its frame count divided by 10; it is not an integrity certificate for the original recording.

Each frame launches external tools and writes intermediates, so this approach suits short creative clips rather than real-time playback. Start with a readable grid and a few seconds of footage; extend the clip after the first and last frames show the effect you want.