Automating SFTP file polling with Ruby

Kevin van Zonneveld

Co-founder · Amsterdam, The Netherlands · Show bio

Use Net::SFTP to poll a trusted drop directory and publish complete files into a local directory. The script below checks the server’s host key, stages downloads before replacing local files, and keeps remote files unless you explicitly enable deletion. It polls all eligible files again after each 60-second wait; it does not remember which files you have already imported.

Prerequisites and setup

This example uses Linux, Ruby 3.4.10 with OpenSSL support, and Bundler 2.6.9. Ruby 3.4 is a maintained release series⁠; Ruby 3.1 and 3.2 are end of life. Have a compiler and Ruby development headers available for the gems’ native extensions.

You need an existing SFTP account whose administrator has authorized your public key and granted read/list access to a drop directory. For server configuration and key authorization, see the OpenSSH server manual⁠. This unattended example uses a dedicated, unencrypted private key protected by filesystem permissions. Obtain the server’s public host key or fingerprint from its administrator through a trusted channel. If you collect a candidate with ssh-keyscan, verify it before adding it to a dedicated known_hosts file; a scan alone does not establish trust. For a nonstandard port, its host entry must use [hostname]:port.

Start in a new directory and save this as Gemfile. These are the versions used here, including Net::SSH 7.3.0’s Ed25519 dependencies⁠:

source 'https://rubygems.org'
gem 'net-sftp', '4.0.0'
gem 'net-ssh', '7.3.0'
gem 'ed25519', '1.4.0'
gem 'bcrypt_pbkdf', '1.1.1'
gem 'base64', '0.3.0'

Install the bundle in that directory, then retain the generated Gemfile.lock with the script:

bundle install

Use a local destination owned exclusively by this importer. Producers must finish uploading under a dot-prefixed name and rename it into place, then leave the published bytes unchanged. This script does not lock remote files or detect a producer modifying a file while it is being read.

Building the file poller

Save the following as sftp_poller.rb beside Gemfile. It imports regular files directly inside REMOTE_DIR, skipping dotfiles, subdirectories, and symbolic links. Each successful download replaces any existing local file with the same name. A failed download leaves the previous local file in place.

require 'net/sftp'
require 'logger'
require 'json'
require 'tempfile'
require 'fileutils'
require 'time' # Time#iso8601, used by the log formatter below

# Configuration constants
SFTP_HOST   = ENV.fetch('SFTP_HOST')
SFTP_USER   = ENV.fetch('SFTP_USER')
SFTP_PORT   = ENV.fetch('SFTP_PORT', 22).to_i
REMOTE_DIR  = ENV.fetch('REMOTE_DIR')
LOCAL_DIR   = ENV.fetch('LOCAL_DIR', './downloads')
SSH_KEY     = ENV.fetch('SSH_KEY_PATH')
KNOWN_HOSTS = ENV.fetch('SSH_KNOWN_HOSTS', File.expand_path('~/.ssh/known_hosts'))

# Removing the remote copy cannot be undone, so it has to be asked for explicitly.
DELETE_AFTER_DOWNLOAD = ENV.fetch('DELETE_AFTER_DOWNLOAD', 'false') == 'true'

# Initialize structured logger
STDOUT.sync = true
logger = Logger.new(STDOUT)
logger.formatter = proc do |severity, datetime, progname, msg|
  JSON.dump(
    timestamp: datetime.iso8601,
    severity: severity,
    message: msg,
    service: 'sftp-poller'
  ) + "\n"
end

def download_file(sftp, remote_file, final_path, logger)
  temp_path = nil
  begin
    # Stage inside the destination directory so the final rename stays on one filesystem. That is
    # what makes it atomic; Tempfile's default directory is usually a different mount.
    temp = Tempfile.create('sftp-download', File.dirname(final_path))
    temp_path = temp.path
    temp.close

    sftp.download!(remote_file, temp_path)
    File.rename(temp_path, final_path)
    temp_path = nil

    logger.info({ action: 'download_complete', file: remote_file, destination: final_path })
    true
  rescue Net::SFTP::StatusException => e
    logger.error({ action: 'download_failed', file: remote_file, error: e.message, code: e.code })
    false
  rescue SystemCallError, IOError => e
    # A local failure (permissions, a full disk) must not take the whole poller down.
    logger.error({ action: 'download_failed', file: remote_file, error: e.message })
    false
  ensure
    File.unlink(temp_path) if temp_path && File.exist?(temp_path)
  end
end

def with_retries(max_attempts: 3, base_delay: 1, logger:)
  attempt = 0
  begin
    attempt += 1
    yield
  rescue Net::SSH::AuthenticationFailed => e
    logger.error({ action: 'authentication_failed', error: e.message })
    raise
  rescue Errno::ECONNREFUSED, Net::SSH::ConnectionTimeout => e
    if attempt < max_attempts
      delay = base_delay * (2 ** (attempt - 1))
      logger.warn({ action: 'retry_attempt', attempt: attempt, delay: delay, error: e.message })
      sleep delay
      retry
    end
    logger.error({ action: 'max_retries_reached', error: e.message })
    raise
  end
end

# Entry names come from the server, so only a plain basename may be joined onto REMOTE_DIR and
# LOCAL_DIR. A name such as `a/../../escaped.txt` is neither hidden nor a directory, yet it reads
# outside REMOTE_DIR, writes outside LOCAL_DIR, and with deletion enabled removes the wrong remote
# file. Reject those names rather than trying to repair them.
def plain_basename?(name)
  return false if name.nil? || name.empty?
  return false if name.include?('/') || name.include?('\\')
  return false if name == '.' || name == '..'

  File.basename(name) == name
end

def poll_sftp(logger)
  with_retries(logger: logger) do
    Net::SFTP.start(
      SFTP_HOST,
      SFTP_USER,
      port: SFTP_PORT,
      keys: [SSH_KEY],
      keys_only: true,
      config: false,
      use_agent: false,
      auth_methods: ['publickey'],
      non_interactive: true,
      timeout: 10,
      # Refuse to connect to a host whose key is not already trusted.
      verify_host_key: :always,
      user_known_hosts_file: KNOWN_HOSTS,
      global_known_hosts_file: []
    ) do |sftp|
      logger.info({ action: 'connection_established', host: SFTP_HOST, directory: REMOTE_DIR })

      sftp.dir.foreach(REMOTE_DIR) do |entry|
        break if @shutdown

        # A leading dot is usually a producer's partial upload, so skip those quietly.
        next if entry.name.start_with?('.')

        unless plain_basename?(entry.name)
          logger.warn({ action: 'entry_rejected', file: entry.name })
          next
        end

        # download! raises on directories, which would otherwise break every future cycle too.
        next unless entry.attributes.file?

        remote_file = File.join(REMOTE_DIR, entry.name)
        local_file  = File.join(LOCAL_DIR, entry.name)

        next unless download_file(sftp, remote_file, local_file, logger)
        next unless DELETE_AFTER_DOWNLOAD

        begin
          sftp.remove!(remote_file)
          logger.info({ action: 'remote_file_removed', file: remote_file })
        rescue Net::SFTP::StatusException => e
          logger.error({ action: 'remove_failed', file: remote_file, error: e.message })
        end
      end
    end
  end
end

# Ensure the local download directory exists
FileUtils.mkdir_p(LOCAL_DIR)

# Set up signal handling for graceful shutdown
@shutdown = false
Signal.trap('TERM') { @shutdown = true }
Signal.trap('INT') { @shutdown = true }

# Main polling loop with graceful shutdown
until @shutdown
  poll_sftp(logger)
  break if @shutdown

  logger.info({ action: 'polling_wait', delay: 60 })
  60.times do
    break if @shutdown
    sleep 1
  end
end

logger.info({ action: 'shutdown_complete' })

Replace the connection details and absolute paths below, then run this from the script’s directory. SSH_KEY_PATH names the private-key file; do not put the key’s contents in an environment variable.

SFTP_HOST=sftp.example.com \
SFTP_PORT=22 \
SFTP_USER=importer \
REMOTE_DIR=/drop \
LOCAL_DIR=/srv/sftp-import/downloads \
SSH_KEY_PATH=/srv/sftp-import/id_ed25519 \
SSH_KNOWN_HOSTS=/srv/sftp-import/known_hosts \
DELETE_AFTER_DOWNLOAD=false \
bundle exec ruby sftp_poller.rb

After a successful scan, JSON logs contain connection_established, a download_complete event for each imported file, and polling_wait. Check the named destination files, including their bytes, before connecting a consumer. A completion event means the local rename succeeded; it does not mean downstream processing or optional remote deletion succeeded. Empty files are valid imports.

Understanding the script

This script implements several important features:

1. Environment-based Configuration: Uses environment variables for sensitive configuration, following security best practices.

2. Structured Logging: Emits one JSON object per event, with the event fields nested under message rather than serialized into a string, so log aggregators can index them.

3. Atomic File Operations: Stages each download next to its destination and renames it into place. The rename is only atomic within a single filesystem, which is why the temporary file is created in LOCAL_DIR instead of /tmp. Consumers must ignore names beginning with sftp-download, which are temporary files; reserve that prefix and do not use it for input names. Only the final filenames are published atomically. Note that Tempfile.create uses mode 0600, and the rename keeps it, so add a File.chmod before the rename if another account has to read what you imported. This only holds if the script owns LOCAL_DIR exclusively: point it at a directory nothing else writes into, and have consumers move files out rather than create subdirectories inside it. Atomicity also stops at the remote end. A producer that rewrites a file while it is being read hands you a mix of two revisions, so have producers upload under a temporary name and rename into REMOTE_DIR once the bytes are there. Atomic replacement is not a power-loss durability guarantee: the script does not call fsync. Ruby’s Tempfile documentation⁠ explains the permissions and explicit cleanup used here.

4. Untrusted Entry Names: plain_basename? rejects anything the server lists that is not a plain filename. Without it, a name such as a/../../escaped.txt passes both the dotfile check and attributes.file?, then escapes LOCAL_DIR once it is joined on, reads a path outside REMOTE_DIR, and, with deletion enabled, removes a file you never asked for.

5. Error Handling: Retries connection refusal and connection timeout up to three attempts, waiting one second and then two seconds. SFTP status errors and local filesystem errors during an individual download are logged and skipped, so the next entry can still be imported. Unknown or changed host keys, authentication failure, a failed directory listing, and a broken SSH connection terminate the process. Fix the cause before restarting; do not bypass host checks.

6. Shutdown: Ctrl+C or SIGTERM requests shutdown. During the polling wait, the script checks that request once per second. During a transfer, it finishes the current file and any opted-in removal, then stops before another entry. The connection timeout limits initial connection setup, not the whole transfer, so a stalled server can delay shutdown. An ordinary exception runs temporary-file cleanup; SIGKILL or a machine crash cannot. After a forced stop, remove leftover sftp-download* files only while the importer is stopped.

7. Optional Remote Cleanup: Removes successfully downloaded files from the remote server to prevent duplicate processing, but only when DELETE_AFTER_DOWNLOAD=true. The removal targets a name, not the revision that was downloaded, so it depends on the same write-once discipline as the warning above. Leaving it off means you need another way to avoid reprocessing, such as a local ledger of imported filenames.

Production deployment

The runnable example above is a foreground process. Before supervising it, decide how consumers avoid duplicate processing, monitor per-file errors as well as process exits, and provide durable storage for downloads. Run only one importer per destination. The following are packaging considerations, not complete deployment configurations.

Using Systemd

The service identity needs read access to the key, trusted-host file, script, and installed bundle, plus write access to the destination. Use the script’s directory as its working directory and invoke it through Bundler. Size the stop grace period for your transfers; a shutdown request does not cancel an active download. A supervisor that eventually kills a stalled importer can leave a temporary file, so recovery must account for that.

Using docker

Use a maintained Ruby image, install the same locked bundle, and keep credentials outside the image. Mount the key and trusted-host file read-only and persist the download directory on a volume. Make sure Ruby receives the stop signal. Docker’s default Linux stop grace is only 10 seconds; after that it sends SIGKILL. Configure the grace period for your workload and test an active transfer, not just an idle container. See Docker’s stop behavior⁠.

Security best practices

1. Host Key Verification:

   • Keep verify_host_key: :always so an unknown or changed host key aborts the connection
   • Provision known_hosts from a fingerprint you confirmed out of band, not from whatever the first connection happens to present
   • Ship known_hosts with the deployment (see the SSH_KNOWN_HOSTS variable above) rather than relying on the running user's home directory

2. SSH Key Management:

   • Rotate SSH keys regularly
   • The pinned bundle includes the optional gems needed for Ed25519 keys in Net::SSH 7.3.0
   • Restrict private-key file access to the importer’s account

3. Network Security:

   • Restrict SFTP access to specific IP ranges
   • Use strong ciphers and key exchange algorithms
   • Monitor transfers that stall; the initial connection timeout is not a transfer deadline

4. File Access:

   • Use minimal permissions for both local and remote files
   • Implement file integrity checks
   • Clean up temporary files properly

5. Monitoring:

   • Set up alerts for failed downloads and connection issues
   • Monitor disk space usage
   • Track processing metrics

Conclusion

This Ruby script is a starting point for automated SFTP file imports. It covers the cases that tend to bite first: host key verification, entry names the server controls, a local write that is either complete or absent, and a failure that must not end the polling loop. Deduplication across restarts, disk space limits, and alerting on a poller that has gone quiet are still yours to add.

If you already use Transloadit, see the SFTP import documentation.