Automate video highlights with Ruby and FFmpeg

Tim Koschützki

Co-founder · Berlin, Germany ·

Finding useful moments in long footage takes time. Ruby can automate scene-change detection, extract candidate clips, and assemble them into a review reel with FFmpeg. This is a visual-change heuristic, not a system that understands which moments are important.

Introduction to automated video highlights

The workflow detects large changes between adjacent frames, selects timestamps far enough apart, and encodes short clips around them. A cut, flash, or slide transition may be a useful candidate, but a goal or spoken insight may have no scene cut. Review the generated reel before publishing it.

Setting up Ruby and FFmpeg

Install Ruby, FFmpeg, and FFprobe. On macOS:

brew install ruby ffmpeg

On Ubuntu/Debian:

sudo apt-get update
sudo apt-get install ruby ffmpeg

The example uses Ruby's standard library rather than a third-party FFmpeg wrapper. Check the tools:

ruby -v
ffmpeg -version
ffprobe -version

Version compatibility

Use a maintained Ruby release on Linux or macOS and an FFmpeg build with the select and metadata filters and the libx264 encoder. This CLI processes one trusted local video at a time. For untrusted uploads, use an isolated worker with filesystem, network, disk, CPU, and wall-clock limits. A decoder process is not a security boundary by itself.

Extracting key moments using FFmpeg filters

The scene score estimates visual differences between frames. For a manual probe in a fresh working directory, this command writes selected frame timestamps to timestamps.txt:

ffmpeg -nostdin -n -i input.mp4 -map 0:v:0 \
  -vf "select='gt(scene,0.3)',metadata=print:file=timestamps.txt" \
  -an -f null -

Keep scene detection and clip extraction on FFmpeg's input-relative timeline. Do not reset the video stream's timestamps independently: audio may start before the first video frame. Detection and extraction both select the first video track explicitly. The filter's text file is separate from FFmpeg's regular output-file checks: do not reuse an existing timestamps.txt. The Ruby implementation creates a private job directory for it.

Implementing Ruby scripts for automation

Save this class as highlight_extractor.rb. It keeps generated filenames fixed and passes input paths as separate process arguments, so paths do not enter shell or filter syntax. A failed stage raises an error rather than silently producing a successful batch report.

require 'fileutils'
require 'json'
require 'open3'

class HighlightExtractor
  def initialize(input_file, output_dir)
    @input = File.expand_path(input_file)
    raise ArgumentError, 'Input must be a regular file' unless File.file?(@input)
    @output = File.expand_path(output_dir)
  end

  def generate(threshold: 0.3, clip_duration: 5.0, pre_roll: 2.0, min_gap: 10.0)
    unless threshold.finite? && (0.0..1.0).cover?(threshold) &&
           clip_duration.finite? && clip_duration > 0 &&
           pre_roll.finite? && pre_roll >= 0 && min_gap.finite? && min_gap >= clip_duration
      raise ArgumentError, 'Invalid scene or clip settings'
    end
    # Refuse earlier job outputs before taking ownership of this directory.
    Dir.mkdir(@output, 0700)
    begin
      duration = video_duration
      selected = detect_scene_changes(threshold).each_with_object([]) do |time, result|
        next if time >= duration
        result << time if result.empty? || time - result.last >= min_gap
      end
      # Cap output count so a noisy video cannot create an unlimited clip set.
      clips = selected.first(20).each_with_index.map do |time, index|
        start = [time - pre_roll, 0.0].max
        length = [clip_duration, duration - start].min
        name = format('highlight_%03d.mp4', index + 1)
        execute('-i', @input, '-ss', start.to_s, '-t', length.to_s,
                '-map', '0:v:0', '-map', '0:a:0?', '-c:v', 'libx264', '-preset', 'fast',
                '-pix_fmt', 'yuv420p', '-c:a', 'aac', '-threads', '2', name)
        name
      end
      create_highlight_compilation(clips) unless clips.empty?
      clips.length
    rescue StandardError
      FileUtils.remove_entry(@output)
      raise
    end
  end

  private

  def video_duration
    output, status = Open3.capture2('ffprobe', '-v', 'error', '-show_entries',
                                   'format=duration', '-of', 'json', @input)
    raise IOError, 'Cannot inspect input video' unless status.success?
    duration = Float(JSON.parse(output).fetch('format').fetch('duration'))
    raise ArgumentError, 'Invalid video duration' unless duration.finite? && duration > 0
    duration
  end

  def execute(*arguments)
    unless system('ffmpeg', '-nostdin', '-n', *arguments, chdir: @output)
      raise IOError, 'FFmpeg processing failed'
    end
  end

  def detect_scene_changes(threshold)
    execute('-i', @input, '-map', '0:v:0', '-vf',
            "select='gt(scene,#{threshold})',metadata=print:file=timestamps.txt",
            '-an', '-f', 'null', '-')
    File.foreach(File.join(@output, 'timestamps.txt')).each_with_object([]) do |line, times|
      match = line.match(/pts_time:([0-9.eE+-]+)/)
      times << Float(match[1]) if match
    end.uniq.sort
  end

  def create_highlight_compilation(clips)
    File.write(File.join(@output, 'filelist.txt'), clips.map { |name| "file '#{name}'\n" }.join)
    execute('-f', 'concat', '-safe', '1', '-i', 'filelist.txt', '-c', 'copy', 'highlight_reel.mp4')
  end
end

The concat list contains only generated basenames, so FFmpeg's safe-path mode stays enabled. All clips come from the same source and encoder settings. Audio is optional, but this example is not a general-purpose concatenator for unrelated files with different stream layouts.

The output-side -ss option decodes and discards footage before each clip's start time. This straightforward decoding path does not depend on usable seek indexes, but repeats work for late highlights. For long, indexed recordings, benchmark input-side accurate seeking and verify clip boundaries and audio/video offsets before changing that behavior.

Save this caller beside the class as generate_highlights.rb:

require_relative 'highlight_extractor'

abort 'Usage: ruby generate_highlights.rb <input-video> <new-output-directory>' unless ARGV.length == 2
begin
  count = HighlightExtractor.new(*ARGV).generate
  puts(count.zero? ? 'No scene changes selected' : "Created #{count} candidate highlights")
rescue StandardError
  warn 'Highlight generation failed. Check the input, destination, and FFmpeg installation.'
  exit 1
end

Run it from your terminal:

ruby generate_highlights.rb my_video.mp4 my_highlights_folder

An existing output directory is refused and remains untouched. A failure after creating the job directory removes that job's partial outputs. A valid video with no selected scene changes succeeds with a clear message and no compilation file. Local FFmpeg diagnostics can include filenames; do not forward them to web clients or use sensitive filenames in shared logs.

Performance optimization

Process a short representative clip before a long recording.
Limit concurrent workers and monitor storage use. The twenty-clip cap bounds generated clips, not the input duration or scene-analysis work.
Apply a wall-clock limit to the whole worker and terminate its child processes on cancellation. The example does not implement a process supervisor or promise per-job timeout enforcement.
Choose hardware encoders only after validating quality and stream compatibility; a faster encoder can require different flags and output settings.

Optimizing highlight selection

A lower scene threshold selects more changes, including noise; a higher threshold can miss subtle transitions. The min_gap setting keeps selected timestamps at least that many seconds apart and must be at least clip_duration. Selection is chronological, not a ranking of the most exciting moments. Adjust pre-roll and duration after reviewing your content.

Practical examples and use cases

Scene-change candidates can help review edited interviews, slide transitions, and montage footage. Do not assume scene detection recognizes goals, wins, specific speakers, or spoken conclusions. Those tasks require additional signals and separate validation. The generated reel is a starting point for editorial review.

Conclusion and potential enhancements

Ruby can orchestrate FFmpeg's documented interfaces without a wrapper dependency, keep job outputs separate, and surface failed stages. Add audio analysis or a validated event detector only when scene changes are insufficient for your use case.

For managed processing, Transloadit's video encoding service provides FFmpeg-based Robots for encoding and media workflows.