Import files from Cloudflare R2 in Java with Rclone

Kevin van Zonneveld

Co-founder · Amsterdam, The Netherlands ·

Cloudflare R2 provides developers with a cost-effective, performant, and reliable object storage solution. Integrating it into your Java applications can significantly streamline your file management workflows. In this DevTip, we'll explore how to efficiently import files from Cloudflare R2 using the powerful open-source tool Rclone.

Introduction to Cloudflare R2

Cloudflare R2 is an S3-compatible object storage service designed to eliminate egress fees, making it ideal for applications requiring frequent data retrieval. Its compatibility with the S3 API simplifies integration with existing tools and workflows used for file importing.

Overview of Rclone

Rclone is an open-source command-line tool that synchronizes files and directories to and from various cloud storage providers. It supports numerous storage backends, including Cloudflare R2, and provides robust features such as syncing, copying, and mounting remote storage. It's one of the most popular open-source tools for cloud storage management.

Setting up Rclone for Cloudflare R2

First, install Rclone if you haven't already. You can usually do this with a single command. After installation, verify it's working:

# Install Rclone (linux/macos/bsd)
curl -fsSL --retry 3 https://rclone.org/install.sh | sudo bash

# Verify installation
rclone version

For other operating systems or methods, refer to the official Rclone installation guide.

Next, configure Rclone to connect to your Cloudflare R2 bucket using the interactive configuration tool:

# Configure Rclone (interactive)
rclone config

Follow the interactive prompts:

Choose n for a new remote.
Enter a name for your remote (e.g., cloudflare_r2).
Select s3 (or the corresponding number) as the storage type.
For the provider, select Cloudflare (or the corresponding number).
Choose Enter credentials value here (usually option 1) or let Rclone find credentials if configured elsewhere (e.g., environment variables).
Provide your Cloudflare R2 Access Key ID.
Provide your Cloudflare R2 Secret Access Key.
Set the Endpoint URL for your R2 bucket: https://<accountid>.r2.cloudflarestorage.com (replace <accountid> with your actual Cloudflare account ID).
Set region to auto. R2 buckets are distributed across Cloudflare's network, so auto is the value Rclone's own Cloudflare walkthrough offers first. Leave Location constraint, which is a separate S3 option, blank.
Set the ACL (Access Control List). private is a common and secure choice.
Review the advanced configuration options (defaults are often fine) and save the configuration.

Your resulting configuration in the Rclone config file (~/.config/rclone/rclone.conf by default) should look similar to this:

[cloudflare_r2]
type = s3
provider = Cloudflare
access_key_id = YOUR_ACCESS_KEY_ID
secret_access_key = YOUR_SECRET_ACCESS_KEY
endpoint = https://<accountid>.r2.cloudflarestorage.com
region = auto
acl = private

Remember to replace the placeholder values with your actual credentials and account ID, and ensure this configuration file is appropriately secured.

Integrating Rclone with Java applications

Java applications can invoke Rclone commands using the ProcessBuilder class. This allows you to leverage Rclone's capabilities directly within your Java code. Here's a practical example demonstrating how to import files from Cloudflare R2. The examples require Java 17 or newer:

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.nio.charset.CodingErrorAction;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.time.Duration;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.concurrent.TimeUnit;

public class RcloneImporter {

    /** Rclone writes a lot on a large transfer, so only this many lines are kept for diagnostics. */
    private static final int MAX_REPORTED_LINES = 200;

    /** What Rclone exited with, and the (capped) output it produced. */
    public record RcloneResult(int exitCode, List<String> lines) {
        public boolean succeeded() {
            return exitCode == 0;
        }
    }

    /**
     * Runs an Rclone command, enforcing a wall-clock timeout, and returns its result.
     *
     * <p>Output is redirected to a file rather than read from a pipe. Draining a pipe on the calling
     * thread has to finish before {@code waitFor} is even reached, so a hung Rclone would block in
     * {@code readLine()} forever and the timeout would never fire.
     *
     * @param builder The command and optional environment, configured by the caller.
     * @param timeout How long to let the command run before killing it.
     * @throws IOException          If the process cannot be started or has to be killed.
     * @throws InterruptedException If this thread is interrupted while waiting.
     */
    private static RcloneResult runRclone(ProcessBuilder builder, Duration timeout)
            throws IOException, InterruptedException {
        Path output = Files.createTempFile("rclone-", ".log");
        try {
            builder.redirectErrorStream(true);
            builder.redirectOutput(output.toFile());
            Process process = builder.start();
            try {
                if (!process.waitFor(timeout.toMillis(), TimeUnit.MILLISECONDS)) {
                    throw new IOException("Rclone timed out after " + timeout);
                }
                return new RcloneResult(process.exitValue(), readCapped(output));
            } finally {
                // Finish cleanup even if the caller interrupts while waiting for termination.
                process.destroyForcibly();
                boolean interrupted = false;
                while (process.isAlive()) {
                    try {
                        process.waitFor();
                    } catch (InterruptedException e) {
                        interrupted = true;
                    }
                }
                if (interrupted) Thread.currentThread().interrupt();
            }
        } finally {
            Files.deleteIfExists(output);
        }
    }

    private static List<String> readCapped(Path output) throws IOException {
        // Diagnostics may contain non-UTF-8 filename bytes; they must not change transfer status.
        var decoder = StandardCharsets.UTF_8.newDecoder()
            .onMalformedInput(CodingErrorAction.REPLACE)
            .onUnmappableCharacter(CodingErrorAction.REPLACE);
        try (var reader = new BufferedReader(new InputStreamReader(Files.newInputStream(output), decoder))) {
            return reader.lines().limit(MAX_REPORTED_LINES).toList();
        }
    }

    /**
     * Imports files from a specified Cloudflare R2 path to a local path using Rclone.
     *
     * @param remoteName The name of the configured Rclone remote (e.g., "cloudflare_r2").
     * @param remotePath The path within the R2 bucket (e.g., "my-bucket/path/to/files").
     * @param localPath  The local directory path where files will be downloaded.
     * @param timeout    How long the copy may run before Rclone is killed.
     */
    public static void importFiles(String remoteName, String remotePath, String localPath,
            Duration timeout) throws IOException, InterruptedException {
        List<String> command = new ArrayList<>(List.of(
            "rclone",
            "copy", // Keeps destination-only files; "sync" would delete them
            remoteName + ":" + remotePath, // Format: remote:path/to/dir
            localPath // Destination local directory
        ));

        // Example: add flags for parallel transfers
        // command.addAll(List.of("--transfers", "8"));

        RcloneResult result = runRclone(new ProcessBuilder(command), timeout);
        for (String line : result.lines()) {
            // Replace with proper logging in a real application. Rclone echoes remote paths and
            // the endpoint, so treat these lines as diagnostics rather than user-facing output.
            System.out.println("Rclone Output: " + line);
        }
        if (!result.succeeded()) {
            throw new IOException("Rclone exited with code " + result.exitCode());
        }
    }

    public static void main(String[] args) {
        String rcloneRemoteName = "cloudflare_r2"; // Matches the name used in `rclone config`
        String bucketPath = "my-data-bucket/source-files"; // Path inside your R2 bucket
        String localDirectory = "./downloaded-files"; // Local destination directory

        try {
            Files.createDirectories(Path.of(localDirectory));
            System.out.println("Starting file import from Cloudflare R2...");
            importFiles(rcloneRemoteName, bucketPath, localDirectory, Duration.ofMinutes(10));
            System.out.println("File import completed successfully.");
        } catch (IOException e) {
            // Handle the error appropriately in your application
            System.err.println("Rclone import failed: " + e.getMessage());
        } catch (InterruptedException e) {
            // Only an actual interruption should restore the interrupt flag
            Thread.currentThread().interrupt();
            System.err.println("Rclone import was interrupted");
        }
    }

}

This improved Java implementation executes the Rclone command to copy files from your Cloudflare R2 bucket to a local directory. It includes better process output handling, timeout management, and more robust error checking. Use a dedicated destination and inspect a --dry-run before importing valuable data: copy can replace changed destination files, while sync additionally deletes destination-only files.

Common issues and troubleshooting tips

When integrating Rclone with Java for Cloudflare R2 operations, you might encounter these issues:

1. Authentication errors

Incorrect Credentials: Double-check the access_key_id and secret_access_key in your Rclone configuration or environment variables.
Wrong Endpoint: Ensure the endpoint URL is correct and includes your specific Cloudflare account ID (https://<accountid>.r2.cloudflarestorage.com).
Permissions: Verify that the R2 API token associated with your credentials has the necessary permissions (e.g., Object Read only) for the target bucket and objects.
ACL Settings: Ensure the acl setting in your Rclone config (private, public-read, etc.) aligns with your bucket policy and access needs.

2. Network issues

Firewall Restrictions: Ensure your server's firewall allows outbound HTTPS connections (port 443) to the Cloudflare R2 endpoint (*.r2.cloudflarestorage.com).
Connectivity: Verify general network connectivity from the machine running the Java application to Cloudflare's services (e.g., using ping or curl).

3. Performance optimization

Parallel Transfers: Use the --transfers N flag (e.g., --transfers 8) in your Rclone command to perform multiple file transfers concurrently, significantly speeding up operations with many small files. Add this to the command list in the Java code.
Large File Downloads: --s3-chunk-size tunes multipart uploads, so it does nothing for the import direction. For large downloads use --multi-thread-streams N together with --multi-thread-cutoff SIZE, which splits a single large file across several connections.
Bandwidth Limit: If needed, use --bwlimit RATE (e.g., --bwlimit 10M for 10 MiB/s, measured in bytes rather than bits) to control bandwidth usage.

4. Java-specific issues

Rclone Not Found: Ensure the rclone executable is in the system's PATH environment variable accessible by the Java process, or provide the full path to the executable in the ProcessBuilder command list.
Process Handling: Never read a subprocess's output on the thread that also has to enforce the timeout. Redirect the output to a file (as shown above) or drain it on a separate thread; otherwise a stalled Rclone blocks the reader and the timeout is unreachable. redirectErrorStream(true) keeps stderr in the same stream.
Timeouts: Call process.waitFor(timeout, unit) before touching the output, and follow a timed-out destroyForcibly() with a plain waitFor() so the process is really gone before you move on. Adjust the timeout duration based on expected operation time.
Resource Cleanup: Ensure Process resources are handled correctly, especially in long-running applications. The example redirects output to a temporary file and deletes it in a finally block, and ensuring the process terminates (via waitFor or destroyForcibly) is crucial.

Advanced usage examples

For a readable object listing, use Rclone's lsf command. For machine-readable inventories, use lsjson, parse its JSON output and keep stdout separate from diagnostic stderr. The capped diagnostics above are not an inventory. A successful or nonempty directory listing does not prove that a particular file exists, and S3-compatible stores cannot distinguish a missing prefix from an empty directory. Use the S3-compatible SDK's object-metadata request when you need an exact object-existence check; authentication and network failures must remain errors, not be reported as absence.

Security best practices

When integrating external tools like Rclone and handling cloud credentials in Java applications, prioritize security:

Avoid Hardcoding Credentials: Never embed your Cloudflare R2 Access Key ID or Secret Access Key directly in your source code.
Use Secure Credential Storage:
- Rclone Config File: Let Rclone use its standard configuration file (rclone.conf), but ensure the file itself has restricted read permissions (e.g., chmod 600 ~/.config/rclone/rclone.conf). This is often the simplest approach.
- Environment Variables: Configure Rclone to read credentials from environment variables. A remote defined this way needs its type and endpoint too, not just the keys, or Rclone reports didn't find section in config file: set RCLONE_CONFIG_CLOUDFLARE_R2_TYPE=s3, RCLONE_CONFIG_CLOUDFLARE_R2_PROVIDER=Cloudflare, RCLONE_CONFIG_CLOUDFLARE_R2_ENDPOINT=https://<accountid>.r2.cloudflarestorage.com, RCLONE_CONFIG_CLOUDFLARE_R2_ACCESS_KEY_ID and RCLONE_CONFIG_CLOUDFLARE_R2_SECRET_ACCESS_KEY securely in your deployment environment.
- Secrets Management System: Integrate with a dedicated secrets management tool (like HashiCorp Vault, AWS Secrets Manager, etc.) to fetch credentials at runtime.
Principle of Least Privilege: Ensure the R2 API token used by Rclone has only the minimum permissions required for its tasks (e.g., read-only access if only importing files). Create specific tokens for specific applications.
Input Validation: Sanitize any user-provided paths or parameters used in constructing Rclone commands if applicable, although using ProcessBuilder with a list of arguments (as shown) significantly mitigates command injection risks compared to building a single command string. These examples use trusted application configuration. Rclone still interprets flags and remote syntax, so validate caller-supplied paths against the intended local root and allowed remote and bucket before invoking it.
Error Handling and Logging: Implement robust error handling that logs failures securely. Use a proper logging framework (like Log4j2, SLF4j/Logback) instead of System.out.println or e.printStackTrace() in production. Avoid logging sensitive information like full credentials or detailed internal paths in case of errors.

Here is the environment-variable approach in Java. Rclone reads RCLONE_CONFIG_<REMOTE>_* from the process environment, so the remote is defined without a config file and without secrets ever appearing in argv, where any user on the box could read them out of ps. The fence below is one more method for the RcloneImporter class above, not a file of its own. Place it inside the class; it reuses that class's imports and process runner. This variant uses a separate remote named r2, rather than the earlier cloudflare_r2 remote:

/**
 * Defines an R2 remote purely through Rclone's environment-variable configuration.
 *
 * <p>Rclone needs the type and endpoint as well as the keys. With only the credentials set it
 * reports {@code didn't find section in config file}.
 */
public static void importWithEnvConfig(String remotePath, String localPath)
        throws IOException, InterruptedException {
    String accessKey = System.getenv("R2_ACCESS_KEY_ID");
    String secretKey = System.getenv("R2_SECRET_ACCESS_KEY");
    String endpoint = System.getenv("R2_ENDPOINT");

    if (accessKey == null || secretKey == null || endpoint == null) {
        throw new IllegalStateException("Required R2 environment variables are not set.");
    }

    ProcessBuilder builder = new ProcessBuilder(
        "rclone", "copy", "r2:" + remotePath, localPath);

    Map<String, String> env = builder.environment();
    env.put("RCLONE_CONFIG_R2_TYPE", "s3");
    env.put("RCLONE_CONFIG_R2_PROVIDER", "Cloudflare");
    env.put("RCLONE_CONFIG_R2_REGION", "auto");
    env.put("RCLONE_CONFIG_R2_ENDPOINT", endpoint);
    env.put("RCLONE_CONFIG_R2_ACCESS_KEY_ID", accessKey);
    env.put("RCLONE_CONFIG_R2_SECRET_ACCESS_KEY", secretKey);

    RcloneResult result = runRclone(builder, Duration.ofMinutes(10));
    if (!result.succeeded()) {
        // The log can echo the endpoint, so keep it out of anything user-facing.
        throw new IOException("Rclone exited with code " + result.exitCode());
    }
}

Avoid rclone config create with credentials as arguments: those end up in the process list, and often in shell history and CI logs as well.

Conclusion and additional resources

Integrating Cloudflare R2 with Java using Rclone provides a robust and efficient solution for file importing tasks. The combination offers flexibility, performance, and the cost-effectiveness of R2's zero egress fees for your application's storage needs. By leveraging ProcessBuilder carefully and understanding Rclone's command-line options, you can build powerful cloud storage interactions into your Java services.

For further exploration, check out these resources:

If you're looking for a fully managed solution that handles the complexities of cloud imports, Transloadit offers a dedicated 🤖 Cloudflare Import Robot as part of our File Importing service. This Robot simplifies the process and supports advanced features like recursive directory imports, pagination control, file stub generation for on-demand processing, and secure authentication using Template Credentials. Transloadit also provides a convenient Java SDK to streamline integration with our platform.