Last updated: April 15, 2025

<span aria-hidden="true" id="automate-video-highlights-with-ruby-and-ffmpeg"></span>

# Automate video highlights with Ruby and FFmpeg

![Tim Koschützki](/assets/images/teammates/avatar-tim-kos-1.jpg?dpl=dpl_C6YH6XrtnwbHLJcm1LDKywQn4CB3)

#### Tim Koschützki

Co-founder · Berlin, Germany · Show bio

[](https://x.com/tim%5Fkos)[](https://github.com/tim-kos)

Automating video editing tasks like creating highlight reels can significantly streamline your content creation workflow. Manually scrubbing through footage to find key moments is often tedious. In this DevTip, we'll explore how Ruby and FFmpeg can be combined to automatically generate engaging video highlights, saving you valuable time and enhancing viewer engagement.

<span aria-hidden="true" id="introduction-to-automated-video-highlights"></span>

## Introduction to automated video highlights

Creating video highlights manually involves identifying significant moments, cutting them, and stitching them together. Automation helps by programmatically detecting potential highlights (like scene changes) and extracting short clips around those points. This makes your content more engaging by providing summaries or teasers, and makes your production workflow more efficient.

<span aria-hidden="true" id="setting-up-ruby-and-ffmpeg"></span>

## Setting up Ruby and FFmpeg

First, ensure you have Ruby and FFmpeg installed on your system.

**Install FFmpeg:**

```bash
# On macOS using homebrew
brew install ffmpeg

# On Ubuntu/Debian
sudo apt update && sudo apt install ffmpeg

```

**Install Ruby:**

If you don't have Ruby installed, you can use a version manager like `rbenv` or `rvm`, or install it via your system's package manager.

```bash
# Example using homebrew on macOS
brew install ruby

```

**Install the Ruby FFmpeg Wrapper:**

We recommend using the `instructure/ruby-ffmpeg` gem, which is actively maintained and supports recent FFmpeg versions. Add it to your project's `Gemfile`:

```ruby
# Gemfile
gem 'ffmpeg', git: 'https://github.com/instructure/ruby-ffmpeg'

```

Then run `bundle install`. Alternatively, install it directly:

```bash
gem install ffmpeg -g 'https://github.com/instructure/ruby-ffmpeg'

```

**Verify Installations:**

Check that both tools are correctly installed and accessible in your path:

```bash
ffmpeg -version
ruby -v
gem list ffmpeg

```

<span aria-hidden="true" id="version-compatibility"></span>

## Version compatibility

Ensure you're using compatible versions for the `instructure/ruby-ffmpeg` gem:

* **Ruby:** Version 3.1 or later
* **FFmpeg:** Versions 4, 5, 6, and 7 are supported
* **Operating System:** Cross-platform (Linux, macOS, Windows)

<span aria-hidden="true" id="extracting-key-moments-using-ffmpeg-filters"></span>

## Extracting key moments using FFmpeg filters

FFmpeg provides powerful filters for analyzing video content. The `select` filter combined with the`scene` score can detect significant changes between frames, which often correspond to potential highlights. We can use this to output timestamps of these changes.

Here's an FFmpeg command to detect scenes with a change score greater than 0.3 (you can adjust this threshold) and print the presentation timestamp (PTS) of these frames to a metadata file:

```bash
ffmpeg -i input.mp4 \
  -filter_complex "[0:v]select='gt(scene,0.3)',metadata=print:file=timestamps.txt" \
  -an -f null -

```

This command analyzes `input.mp4`, uses the `select` filter to find frames where the scene change score is above 0.3, and writes metadata (including timestamps) for these frames to `timestamps.txt`. We use `-an -f null -` to discard the video output as we only need the metadata file.

<span aria-hidden="true" id="implementing-ruby-scripts-for-automation"></span>

## Implementing Ruby scripts for automation

Now, let's create a Ruby class to encapsulate the logic for detecting scenes and creating highlight clips using the `ffmpeg` gem and the command above.

```ruby
# Highlight_extractor.rb
require 'ffmpeg'
require 'fileutils'
require 'etc' # Required for Etc.nprocessors
require 'shellwords'

class HighlightExtractor
  attr_reader :media, :output_dir

  def initialize(input_file, output_dir = 'highlights')
    raise "Input file not found: #{input_file}" unless File.exist?(input_file)
    @media = FFMPEG::Media.new(input_file)
    @output_dir = output_dir
    FileUtils.mkdir_p(@output_dir) # Create output directory if it doesn't exist
  end

  # Detects scene changes and returns an array of timestamps (in seconds)
  def detect_scene_changes(threshold = 0.3)
    timestamp_file = File.join(@output_dir, "timestamps.txt")
    # Configure FFmpeg global settings for performance
    FFMPEG.threads = [Etc.nprocessors, 4].min # Use available processors, max 4

    # Use Shellwords.escape for safety with file paths
    escaped_input_path = Shellwords.escape(media.path)
    escaped_timestamp_file = Shellwords.escape(timestamp_file)

    command = %Q{ffmpeg -threads #{FFMPEG.threads} -i #{escaped_input_path} \
      -filter_complex "[0:v]select='gt(scene,#{threshold})',metadata=print:file=#{escaped_timestamp_file}" \
      -an -f null -}

    puts "Running scene detection command..."
    # puts command # Uncomment for debugging
    system(command) # Execute the FFmpeg command

    # Parse timestamps from the metadata file
    timestamps = []
    if File.exist?(timestamp_file)
      File.readlines(timestamp_file).each do |line|
        match = line.match(/pts_time:([\d.]+)/
        timestamps << match[1].to_f if match
      end
      # FileUtils.rm(timestamp_file) # Optional: Clean up the timestamp file
    else
      puts "Warning: Timestamp file not found at #{timestamp_file}. Scene detection might have failed."
    end

    timestamps.uniq.sort # Return unique, sorted timestamps
  end

  # Creates short video clips around the detected timestamps
  def create_highlight_clips(timestamps, clip_duration = 5, pre_roll = 2)
    highlight_files = []

    timestamps.each_with_index do |time, index|
      start_time = [time - pre_roll, 0].max # Start 'pre_roll' seconds before the scene change, but not before 0
      output_file = File.join(@output_dir, "highlight_#{index + 1}.mp4")

      puts "Creating highlight #{index + 1} starting at #{start_time.round(2)}s..."

      # Define preset for the highlight clip
      preset = FFMPEG::Preset.new(name: 'highlight', filename: '%<basename>s.mp4') do
        video_codec_name 'libx264' # Common H.264 codec
        audio_codec_name 'aac'     # Common AAC audio codec
        seek_time start_time       # Start time for the clip
        duration clip_duration    # Duration of the clip
        # Add other options like bitrate, resolution if needed
        # video_bitrate '1000k'
        # resolution '1280x720'
      end

      transcoder = FFMPEG::Transcoder.new(
        name: "highlight_#{index + 1}",
        presets: [preset]
      )

      begin
        # Configure timeout settings
        FFMPEG.timeout = 300 # 5 minutes global timeout
        FFMPEG.io_timeout = 30 # 30 seconds IO timeout

        transcoder.process(media, output_file)
        highlight_files << output_file
        puts "Successfully created #{output_file}"
      rescue FFMPEG::Error => e
        puts "Error creating highlight #{index + 1} at #{time}s: #{e.message}"
      rescue StandardError => e
        puts "Unexpected error creating highlight #{index + 1}: #{e.message}"
      end
    end

    highlight_files
  end
end

```

Here's how you can use this class in a script:

```ruby
#!/usr/bin/env ruby
# Generate_highlights.rb
require_relative 'highlight_extractor' # Assuming the class is in this file
require 'shellwords'

# --- helper functions ---
def filter_timestamps(timestamps, min_gap)
  return [] if timestamps.empty?
  filtered = [timestamps.first]
  timestamps.drop(1).each do |time|
    filtered << time if (time - filtered.last) >= min_gap
  end
  filtered
end

def create_highlight_compilation(highlight_files, output_file, temp_dir)
  return if highlight_files.empty?

  list_file = File.join(temp_dir, "filelist.txt")
  # Ensure we use relative paths from the perspective of the temp_dir for concat
  begin
    File.open(list_file, 'w') do |f|
      highlight_files.each do |file|
        # Use relative path from the output directory for the list file
        relative_path = File.basename(file)
        f.puts "file '#{relative_path.gsub("'", "'\\\\'"')}'" # Escape single quotes
      end
    end

    # Concatenate all highlights using the concat demuxer
    # Use -safe 0 as we are controlling the paths in filelist.txt
    # Use -c copy for speed (no re-encoding), requires clips to have compatible streams
    escaped_list_file = Shellwords.escape(list_file)
    escaped_output_file = Shellwords.escape(output_file)

    # Run ffmpeg from the output directory to resolve relative paths easily
    concat_command = %Q{ffmpeg -f concat -safe 0 -i #{escaped_list_file} -c copy #{escaped_output_file}}

    puts "\nRunning concatenation command in dir '#{temp_dir}'..."
    # puts concat_command # Uncomment for debugging
    system(concat_command, chdir: temp_dir) # Run ffmpeg in the output directory

    if $?.success?
      puts "Successfully created compilation: #{output_file}"
    else
      puts "Error creating compilation file."
    end
  ensure
    # FileUtils.rm(list_file) if File.exist?(list_file) # Optional: Clean up the list file
  end
end

# --- configuration ---
input_file = ARGV[0]
output_dir = ARGV[1] || 'highlights'
scene_threshold = 0.3 # Sensitivity for scene detection (0.0 to 1.0)
clip_duration = 5     # Duration of each highlight clip in seconds
pre_roll = 2          # Seconds before the detected scene change to start the clip
min_gap_between_highlights = 10 # Minimum seconds between highlights

# --- main execution ---
if input_file.nil?
  puts "Usage: ruby generate_highlights.rb <input_video.mp4> [output_directory]"
  exit(1)
end

unless File.exist?(input_file)
  puts "Error: Input file '#{input_file}' not found."
  exit(1)
end

begin
  extractor = HighlightExtractor.new(input_file, output_dir)

  puts "Detecting scene changes in '#{input_file}' (threshold: #{scene_threshold})..."
  timestamps = extractor.detect_scene_changes(scene_threshold)

  if timestamps.empty?
    puts "No significant scene changes detected."
  else
    puts "Found #{timestamps.size} potential highlight timestamps."

    # Optional: Filter timestamps that are too close together
    filtered_timestamps = filter_timestamps(timestamps, min_gap_between_highlights)
    if timestamps.size != filtered_timestamps.size
        puts "Filtered down to #{filtered_timestamps.size} highlights (min gap: #{min_gap_between_highlights}s)."
    end

    if filtered_timestamps.empty?
        puts "No highlights remaining after filtering."
    else
        puts "\nCreating #{filtered_timestamps.size} highlight clips (duration: #{clip_duration}s, pre-roll: #{pre_roll}s)..."
        highlight_files = extractor.create_highlight_clips(filtered_timestamps, clip_duration, pre_roll)

        if highlight_files.empty?
          puts "\nNo highlight clips were successfully generated."
        else
          puts "\nSuccessfully created #{highlight_files.size} highlight clips in '#{output_dir}/'."

          # Optional: Create a compilation reel
          compilation_file = File.join(output_dir, "highlight_reel.mp4")
          create_highlight_compilation(highlight_files, compilation_file, output_dir)
        end
    end
  end

rescue FFMPEG::Error => e
  puts "\nFFmpeg Error: #{e.message}"
  exit(1)
rescue StandardError => e
  puts "\nAn unexpected error occurred: #{e.message}"
  puts e.backtrace.join("\n")
  exit(1)
end

puts "\nHighlight generation process finished."

```

Save the class as `highlight_extractor.rb` and the script as `generate_highlights.rb`. Run it from your terminal like this:

```bash
ruby generate_highlights.rb my_video.mp4 my_highlights_folder

```

<span aria-hidden="true" id="performance-optimization"></span>

## Performance optimization

Processing video can be resource-intensive. Here are ways to optimize performance:

* **Threading:** Use multiple CPU cores during FFmpeg operations. The `instructure/ruby-ffmpeg` gem allows global configuration:

```ruby
require 'etc'  
# Use up to 4 cores, or fewer if the system has less  
FFMPEG.threads = [Etc.nprocessors, 4].min  
```

* **Timeouts:** Prevent FFmpeg processes from running indefinitely if they encounter issues:

```ruby
FFMPEG.timeout = 300 # Set global timeout to 5 minutes  
FFMPEG.io_timeout = 30 # Set IO timeout to 30 seconds  
```

* **Hardware Acceleration:** For significant speedups on supported hardware (like NVIDIA GPUs with NVENC or macOS with VideoToolbox), explore FFmpeg's hardware acceleration options (e.g.,`-hwaccel cuda`, `-c:v h264_nvenc`). Integrating these often requires specific FFmpeg builds and adds complexity to your commands and presets.

<span aria-hidden="true" id="optimizing-highlight-selection"></span>

## Optimizing highlight selection

* **Threshold Adjustment:** The `scene` threshold (e.g., `0.3`) is crucial. A lower value detects more changes (potentially noise), while a higher value detects only major changes. Experiment to find what works best for your content type.
* **Filtering Logic:** The `filter_timestamps` function in the example script prevents creating highlights too close together. You could enhance this logic, perhaps by considering the \_duration\_between scene changes or analyzing the *magnitude* of the scene change score.

<span aria-hidden="true" id="practical-examples-and-use-cases"></span>

## Practical examples and use cases

This automated approach is valuable in various scenarios:

* **Sports:** Generate highlight reels from game footage by detecting goals, scores, or significant plays (scene changes often accompany these).
* **Webinars/Talks:** Extract key moments or slide transitions for promotional clips.
* **Gaming Streams:** Automatically capture exciting moments like kills, wins, or major events.
* **Interviews:** Identify segments where speakers change or key points are made.
* **Vlogs:** Quickly create montages or summaries from longer footage.
* **Security Footage:** Isolate segments with significant motion or changes.

The compilation example (`create_highlight_compilation`) shows how to easily stitch the generated clips into a single highlight reel using FFmpeg's `concat` demuxer.

<span aria-hidden="true" id="conclusion-and-potential-enhancements"></span>

## Conclusion and potential enhancements

Combining Ruby's scripting capabilities with FFmpeg's powerful media processing tools provides an effective way to automate video highlight generation, significantly boosting content creation efficiency.

Potential enhancements include:

* **Advanced Scene Detection:** Explore other FFmpeg filters (like `blackdetect` or `silencedetect`) or combinations for more nuanced detection.
* **Audio Analysis:** Incorporate audio analysis (e.g., detecting loud sounds, specific keywords, or changes in volume) to identify highlights based on sound events.
* **Machine Learning:** Integrate ML models for more sophisticated content analysis (e.g., object recognition, action detection) to identify specific types of highlights relevant to your domain.
* **Web Interface:** Build a simple web UI (e.g., using Sinatra or Rails) to allow users to upload videos and manage the highlight generation process.

For robust, scalable video processing in production environments, consider cloud-based services. Transloadit leverages FFmpeg extensively in its [video encoding service](/services/video-encoding.md), providing managed infrastructure and pre-built Robots that simplify complex tasks like automated highlight generation within your application workflows.

\#ruby#ffmpeg#video-highlights#automation#content-creation#video-encoding-service

### 👩‍💻 Join 20k+ developers

Sign up for our [monthly newsletter](/newsletters.md) to receive direct links to 3 exclusive tech — and 2 product updates. No less, no more.

Your email:

Get access

## File uploading and encoding. Made simple.

Transloadit streamlines file handling for developers, trusted by brands like Coursera and The New York Times. We’re known for a reliable API, top-notch support, and a strong commitment to open source, with projects like [Uppy⁠](https://uppy.io) and [Tus⁠](https://tus.io) setting standards in file processing.

[Sign up](/c/)[Book a Demo](https://survey.typeform.com/to/kRg47Xi5)

No credit card needed · 5 GB included in the free plan

Cancel anytime