Last updated: September 11

Filter files in Ruby: a practical guide

Kevin van Zonneveld

Co-founder · Amsterdam, The Netherlands ·

Efficient file filtering is essential for everything from log processing to media management. Ruby provides robust tools for file selection that go beyond simple pattern matching. Let's explore practical techniques that scale from basic to advanced scenarios.

Requirements

This guide requires Ruby 2.5 or later for Dir.children. The examples presented use Ruby 3.2.3 and mime-types 3.6.0. For MIME type detection, you'll need the mime-types gem:

# Add to your Gemfile
gem 'mime-types', '3.6.0'

Or install the same version directly:

gem install mime-types -v 3.6.0

Note

The mime-types gem employs modified semantic versioning to track both API changes and registry data updates. For further details, refer to the mime-types documentation.

Core filtering methods

Ruby's standard library offers several immediate solutions for file filtering:

# Filter by extension
docs_dir = '/docs'
pdf_files = Dir.children(docs_dir).select do |f|
  File.file?(File.join(docs_dir, f)) && File.extname(f) == '.pdf'
end

# Filter by size (1MB threshold)
large_files = Dir.glob('*').select do |f|
  begin
    File.file?(f) && File.size(f) > 1_000_000
  rescue Errno::ENOENT, Errno::EACCES => e
    warn "Error accessing #{f}: #{e.message}"
    false
  end
end

# Filter by modification time (last 24 hours)
recent_files = Dir.glob('*').select do |f|
  begin
    File.file?(f) && File.mtime(f) > (Time.now - 86400)
  rescue Errno::ENOENT, Errno::EACCES => e
    warn "Error accessing #{f}: #{e.message}"
    false
  end
end

These methods use the File class utilities for quick checks without loading file contents.

Advanced pattern matching with glob

Ruby's Dir.glob supports UNIX-style pattern matching with some Ruby-specific enhancements:

# Match nested Markdown files
markdown_files = Dir.glob('**/*.md').select { |f| File.file?(f) }

# Match names containing a January 2024 date (not modification times)
jan_files = Dir.glob('*').grep(/(2024-01-\d{2})/).select { |f| File.file?(f) }

# Combined size and type filter
big_images = Dir.glob('*.{jpg,png}').select do |f|
  begin
    File.file?(f) && File.size(f) > 500_000
  rescue Errno::ENOENT, Errno::EACCES => e
    warn "Error accessing #{f}: #{e.message}"
    false
  end
end

Use double star (**) for recursive directory traversal and brace expansion for multiple extensions.

MIME type lookup

Use the mime-types gem to look up the media type associated with a filename extension:

require 'mime/types'

def media_files(dir)
  Dir.children(dir).select do |f|
    begin
      next unless File.file?(File.join(dir, f))
      mime = MIME::Types.type_for(f).first
      mime&.media_type == 'image' || mime&.media_type == 'video'
    rescue StandardError => e
      warn "Error processing #{f}: #{e.message}"
      false
    end
  end
end

# Usage:
visual_assets = media_files('/content/assets')

This approach uses an extension registry; it does not inspect file contents or validate that a file is safe. Use content inspection separately when accepting untrusted uploads. For additional details, see the mime-types gem documentation.

Metadata filtering

Combine multiple metadata points for precise selection:

def recent_documents(path)
  Dir.glob("#{path}/*").select do |f|
    begin
      next unless File.file?(f)

      ext = File.extname(f).downcase
      size = File.size(f)
      modified = File.mtime(f)

      (ext == '.pdf' || ext == '.docx') &&
        size.between?(10_000, 5_000_000) &&
        modified > (Time.now - 7*86400)
    rescue StandardError => e
      warn "Error processing #{f}: #{e.message}"
      false
    end
  end
end

This selects PDF/DOCX files modified in the last week between 10 KB and 5 MB.

Performance considerations

When processing large directories:

Lazy Evaluation: Defer metadata checks with lazy. Dir.glob still builds its result array.

Dir.glob('**/*').lazy
  .select { |f| File.file?(f) && File.size(f) > 1_000_000 }
  .first(10)

Early Exit: Fail fast with break when possible

log_dir = '/logs'
Dir.children(log_dir).each do |name|
  f = File.join(log_dir, name)
  begin
    next unless File.file?(f) && f.end_with?('.log')
    break if File.size(f) > 1_000_000_000  # Stop at first huge log
    process_log(f)
  rescue StandardError => e
    warn "Error processing #{f}: #{e.message}"
  end
end

Define process_log(path) for your application's processing before running this excerpt.

Metadata Caching: Store frequently accessed data

file_cache = {}
Dir.glob('*').each do |f|
  begin
    file_cache[f] = {
      mtime: File.mtime(f),
      size: File.size(f)
    }
  rescue StandardError => e
    warn "Error caching #{f}: #{e.message}"
  end
end

Production-grade example

Here's a complete filtering module with error handling:

require 'mime/types'

class FileFilter
  def initialize(root_dir)
    @root = root_dir
  end

  def find_files(extensions: [], min_size: 0, max_age: Float::INFINITY)
    Dir.glob(File.join(@root, '**', '*')).lazy.select do |path|
      begin
        next unless File.file?(path)

        valid_extension = extensions.empty? || extensions.include?(File.extname(path))
        valid_size = File.size(path) >= min_size
        valid_age = (Time.now - File.mtime(path)) < max_age

        valid_extension && valid_size && valid_age
      rescue StandardError => e
        warn "Error processing #{path}: #{e.message}"
        false
      end
    end
  end
end

# Usage:
filter = FileFilter.new('/user/uploads')
recent_images = filter.find_files(
  extensions: ['.jpg', '.png'],
  min_size: 100_000,
  max_age: 3600 # 1 hour
).first(100)

These examples illustrate practical ways to filter files in Ruby with robust error handling. For more advanced file processing solutions, consider exploring Transloadit's API. Visit our documentation for additional details.

Handling file encodings

Ruby typically handles file name encodings automatically when using UTF-8. However, if you encounter issues with non-ASCII characters in file names, create a UTF-8 display value as follows:

# Convert from Ruby's known source encoding and replace invalid bytes for display
utf8_labels = Dir.glob('*').map do |f|
  f.encode('UTF-8', invalid: :replace, undef: :replace).scrub
end

Keep the original path for filesystem operations. Replacement characters can change a filename; these labels are for display only. force_encoding merely relabels bytes and does not transcode.

Conclusion

Throughout this guide, we explored practical techniques to filter files in Ruby, from simple extension checks and glob patterns to MIME type detection and metadata filtering. By combining these methods, you can build efficient file processing pipelines tailored to your needs. For large-scale, production-ready solutions, consider exploring the robust file processing API offered by Transloadit. Visit our documentation to learn more.