Automating video editing tasks like creating highlight reels can significantly streamline your content creation workflow. Manually scrubbing through footage to find key moments is often tedious. In this DevTip, we'll explore how Ruby and FFmpeg can be combined to automatically generate engaging video highlights, saving you valuable time and enhancing viewer engagement.

Introduction to automated video highlights

Creating video highlights manually involves identifying significant moments, cutting them, and stitching them together. Automation helps by programmatically detecting potential highlights (like scene changes) and extracting short clips around those points. This makes your content more engaging by providing summaries or teasers, and makes your production workflow more efficient.

Setting up Ruby and FFmpeg

First, ensure you have Ruby and FFmpeg installed on your system.

Install FFmpeg:

# On macOS using homebrew
brew install ffmpeg

# On Ubuntu/Debian
sudo apt update && sudo apt install ffmpeg

Install Ruby:

If you don't have Ruby installed, you can use a version manager like rbenv or rvm, or install it via your system's package manager.

# Example using homebrew on macOS
brew install ruby

Install the Ruby FFmpeg Wrapper:

We recommend using the instructure/ruby-ffmpeg gem, which is actively maintained and supports recent FFmpeg versions. Add it to your project's Gemfile:

# Gemfile
gem 'ffmpeg', git: 'https://github.com/instructure/ruby-ffmpeg'

Then run bundle install. Alternatively, install it directly:

gem install ffmpeg -g 'https://github.com/instructure/ruby-ffmpeg'

Verify Installations:

Check that both tools are correctly installed and accessible in your path:

ffmpeg -version
ruby -v
gem list ffmpeg

Version compatibility

Ensure you're using compatible versions for the instructure/ruby-ffmpeg gem:

  • Ruby: Version 3.1 or later
  • FFmpeg: Versions 4, 5, 6, and 7 are supported
  • Operating System: Cross-platform (Linux, macOS, Windows)

Extracting key moments using FFmpeg filters

FFmpeg provides powerful filters for analyzing video content. The select filter combined with the scene score can detect significant changes between frames, which often correspond to potential highlights. We can use this to output timestamps of these changes.

Here's an FFmpeg command to detect scenes with a change score greater than 0.3 (you can adjust this threshold) and print the presentation timestamp (PTS) of these frames to a metadata file:

ffmpeg -i input.mp4 \
  -filter_complex "[0:v]select='gt(scene,0.3)',metadata=print:file=timestamps.txt" \
  -an -f null -

This command analyzes input.mp4, uses the select filter to find frames where the scene change score is above 0.3, and writes metadata (including timestamps) for these frames to timestamps.txt. We use -an -f null - to discard the video output as we only need the metadata file.

Implementing Ruby scripts for automation

Now, let's create a Ruby class to encapsulate the logic for detecting scenes and creating highlight clips using the ffmpeg gem and the command above.

# Highlight_extractor.rb
require 'ffmpeg'
require 'fileutils'
require 'etc' # Required for Etc.nprocessors
require 'shellwords'

class HighlightExtractor
  attr_reader :media, :output_dir

  def initialize(input_file, output_dir = 'highlights')
    raise "Input file not found: #{input_file}" unless File.exist?(input_file)
    @media = FFMPEG::Media.new(input_file)
    @output_dir = output_dir
    FileUtils.mkdir_p(@output_dir) # Create output directory if it doesn't exist
  end

  # Detects scene changes and returns an array of timestamps (in seconds)
  def detect_scene_changes(threshold = 0.3)
    timestamp_file = File.join(@output_dir, "timestamps.txt")
    # Configure FFmpeg global settings for performance
    FFMPEG.threads = [Etc.nprocessors, 4].min # Use available processors, max 4

    # Use Shellwords.escape for safety with file paths
    escaped_input_path = Shellwords.escape(media.path)
    escaped_timestamp_file = Shellwords.escape(timestamp_file)

    command = %Q{ffmpeg -threads #{FFMPEG.threads} -i #{escaped_input_path} \
      -filter_complex "[0:v]select='gt(scene,#{threshold})',metadata=print:file=#{escaped_timestamp_file}" \
      -an -f null -}

    puts "Running scene detection command..."
    # puts command # Uncomment for debugging
    system(command) # Execute the FFmpeg command

    # Parse timestamps from the metadata file
    timestamps = []
    if File.exist?(timestamp_file)
      File.readlines(timestamp_file).each do |line|
        match = line.match(/pts_time:([\d.]+)/
        timestamps << match[1].to_f if match
      end
      # FileUtils.rm(timestamp_file) # Optional: Clean up the timestamp file
    else
      puts "Warning: Timestamp file not found at #{timestamp_file}. Scene detection might have failed."
    end

    timestamps.uniq.sort # Return unique, sorted timestamps
  end

  # Creates short video clips around the detected timestamps
  def create_highlight_clips(timestamps, clip_duration = 5, pre_roll = 2)
    highlight_files = []

    timestamps.each_with_index do |time, index|
      start_time = [time - pre_roll, 0].max # Start 'pre_roll' seconds before the scene change, but not before 0
      output_file = File.join(@output_dir, "highlight_#{index + 1}.mp4")

      puts "Creating highlight #{index + 1} starting at #{start_time.round(2)}s..."

      # Define preset for the highlight clip
      preset = FFMPEG::Preset.new(name: 'highlight', filename: '%<basename>s.mp4') do
        video_codec_name 'libx264' # Common H.264 codec
        audio_codec_name 'aac'     # Common AAC audio codec
        seek_time start_time       # Start time for the clip
        duration clip_duration    # Duration of the clip
        # Add other options like bitrate, resolution if needed
        # video_bitrate '1000k'
        # resolution '1280x720'
      end

      transcoder = FFMPEG::Transcoder.new(
        name: "highlight_#{index + 1}",
        presets: [preset]
      )

      begin
        # Configure timeout settings
        FFMPEG.timeout = 300 # 5 minutes global timeout
        FFMPEG.io_timeout = 30 # 30 seconds IO timeout

        transcoder.process(media, output_file)
        highlight_files << output_file
        puts "Successfully created #{output_file}"
      rescue FFMPEG::Error => e
        puts "Error creating highlight #{index + 1} at #{time}s: #{e.message}"
      rescue StandardError => e
        puts "Unexpected error creating highlight #{index + 1}: #{e.message}"
      end
    end

    highlight_files
  end
end

Here's how you can use this class in a script:

#!/usr/bin/env ruby
# Generate_highlights.rb
require_relative 'highlight_extractor' # Assuming the class is in this file
require 'shellwords'

# --- helper functions ---
def filter_timestamps(timestamps, min_gap)
  return [] if timestamps.empty?
  filtered = [timestamps.first]
  timestamps.drop(1).each do |time|
    filtered << time if (time - filtered.last) >= min_gap
  end
  filtered
end

def create_highlight_compilation(highlight_files, output_file, temp_dir)
  return if highlight_files.empty?

  list_file = File.join(temp_dir, "filelist.txt")
  # Ensure we use relative paths from the perspective of the temp_dir for concat
  begin
    File.open(list_file, 'w') do |f|
      highlight_files.each do |file|
        # Use relative path from the output directory for the list file
        relative_path = File.basename(file)
        f.puts "file '#{relative_path.gsub("'", "'\\\\'"')}'" # Escape single quotes
      end
    end

    # Concatenate all highlights using the concat demuxer
    # Use -safe 0 as we are controlling the paths in filelist.txt
    # Use -c copy for speed (no re-encoding), requires clips to have compatible streams
    escaped_list_file = Shellwords.escape(list_file)
    escaped_output_file = Shellwords.escape(output_file)

    # Run ffmpeg from the output directory to resolve relative paths easily
    concat_command = %Q{ffmpeg -f concat -safe 0 -i #{escaped_list_file} -c copy #{escaped_output_file}}

    puts "\nRunning concatenation command in dir '#{temp_dir}'..."
    # puts concat_command # Uncomment for debugging
    system(concat_command, chdir: temp_dir) # Run ffmpeg in the output directory

    if $?.success?
      puts "Successfully created compilation: #{output_file}"
    else
      puts "Error creating compilation file."
    end
  ensure
    # FileUtils.rm(list_file) if File.exist?(list_file) # Optional: Clean up the list file
  end
end

# --- configuration ---
input_file = ARGV[0]
output_dir = ARGV[1] || 'highlights'
scene_threshold = 0.3 # Sensitivity for scene detection (0.0 to 1.0)
clip_duration = 5     # Duration of each highlight clip in seconds
pre_roll = 2          # Seconds before the detected scene change to start the clip
min_gap_between_highlights = 10 # Minimum seconds between highlights

# --- main execution ---
if input_file.nil?
  puts "Usage: ruby generate_highlights.rb <input_video.mp4> [output_directory]"
  exit(1)
end

unless File.exist?(input_file)
  puts "Error: Input file '#{input_file}' not found."
  exit(1)
end

begin
  extractor = HighlightExtractor.new(input_file, output_dir)

  puts "Detecting scene changes in '#{input_file}' (threshold: #{scene_threshold})..."
  timestamps = extractor.detect_scene_changes(scene_threshold)

  if timestamps.empty?
    puts "No significant scene changes detected."
  else
    puts "Found #{timestamps.size} potential highlight timestamps."

    # Optional: Filter timestamps that are too close together
    filtered_timestamps = filter_timestamps(timestamps, min_gap_between_highlights)
    if timestamps.size != filtered_timestamps.size
        puts "Filtered down to #{filtered_timestamps.size} highlights (min gap: #{min_gap_between_highlights}s)."
    end

    if filtered_timestamps.empty?
        puts "No highlights remaining after filtering."
    else
        puts "\nCreating #{filtered_timestamps.size} highlight clips (duration: #{clip_duration}s, pre-roll: #{pre_roll}s)..."
        highlight_files = extractor.create_highlight_clips(filtered_timestamps, clip_duration, pre_roll)

        if highlight_files.empty?
          puts "\nNo highlight clips were successfully generated."
        else
          puts "\nSuccessfully created #{highlight_files.size} highlight clips in '#{output_dir}/'."

          # Optional: Create a compilation reel
          compilation_file = File.join(output_dir, "highlight_reel.mp4")
          create_highlight_compilation(highlight_files, compilation_file, output_dir)
        end
    end
  end

rescue FFMPEG::Error => e
  puts "\nFFmpeg Error: #{e.message}"
  exit(1)
rescue StandardError => e
  puts "\nAn unexpected error occurred: #{e.message}"
  puts e.backtrace.join("\n")
  exit(1)
end

puts "\nHighlight generation process finished."

Save the class as highlight_extractor.rb and the script as generate_highlights.rb. Run it from your terminal like this:

ruby generate_highlights.rb my_video.mp4 my_highlights_folder

Performance optimization

Processing video can be resource-intensive. Here are ways to optimize performance:

  • Threading: Use multiple CPU cores during FFmpeg operations. The instructure/ruby-ffmpeg gem allows global configuration:

    require 'etc'
# Use up to 4 cores, or fewer if the system has less
FFMPEG.threads = [Etc.nprocessors, 4].min

  • Timeouts: Prevent FFmpeg processes from running indefinitely if they encounter issues:

    FFMPEG.timeout = 300 # Set global timeout to 5 minutes
FFMPEG.io_timeout = 30 # Set IO timeout to 30 seconds

  • Hardware Acceleration: For significant speedups on supported hardware (like NVIDIA GPUs with NVENC or macOS with VideoToolbox), explore FFmpeg's hardware acceleration options (e.g., -hwaccel cuda, -c:v h264_nvenc). Integrating these often requires specific FFmpeg builds and adds complexity to your commands and presets.

Optimizing highlight selection

  • Threshold Adjustment: The scene threshold (e.g., 0.3) is crucial. A lower value detects more changes (potentially noise), while a higher value detects only major changes. Experiment to find what works best for your content type.
  • Filtering Logic: The filter_timestamps function in the example script prevents creating highlights too close together. You could enhance this logic, perhaps by considering the duration between scene changes or analyzing the magnitude of the scene change score.

Practical examples and use cases

This automated approach is valuable in various scenarios:

  • Sports: Generate highlight reels from game footage by detecting goals, scores, or significant plays (scene changes often accompany these).
  • Webinars/Talks: Extract key moments or slide transitions for promotional clips.
  • Gaming Streams: Automatically capture exciting moments like kills, wins, or major events.
  • Interviews: Identify segments where speakers change or key points are made.
  • Vlogs: Quickly create montages or summaries from longer footage.
  • Security Footage: Isolate segments with significant motion or changes.

The compilation example (create_highlight_compilation) shows how to easily stitch the generated clips into a single highlight reel using FFmpeg's concat demuxer.

Conclusion and potential enhancements

Combining Ruby's scripting capabilities with FFmpeg's powerful media processing tools provides an effective way to automate video highlight generation, significantly boosting content creation efficiency.

Potential enhancements include:

  • Advanced Scene Detection: Explore other FFmpeg filters (like blackdetect or silencedetect) or combinations for more nuanced detection.
  • Audio Analysis: Incorporate audio analysis (e.g., detecting loud sounds, specific keywords, or changes in volume) to identify highlights based on sound events.
  • Machine Learning: Integrate ML models for more sophisticated content analysis (e.g., object recognition, action detection) to identify specific types of highlights relevant to your domain.
  • Web Interface: Build a simple web UI (e.g., using Sinatra or Rails) to allow users to upload videos and manage the highlight generation process.

For robust, scalable video processing in production environments, consider cloud-based services. Transloadit leverages FFmpeg extensively in its video encoding service, providing managed infrastructure and pre-built Robots that simplify complex tasks like automated highlight generation within your application workflows.