Using Rclone to manage files in DigitalOcean Spaces via CLI

Kevin van Zonneveld

Co-founder · Amsterdam, The Netherlands ·

Managing files in cloud storage can be a challenge for developers. With Rclone—a robust, open-source command-line tool—you can effortlessly manage files in DigitalOcean Spaces, a scalable, S3-compatible object storage service. In this DevTip, we explain how to leverage Rclone for listing, uploading, downloading, and synchronizing files, thereby streamlining your workflow.

What is Rclone?

Rclone is an open-source command-line program designed for managing files across various cloud storage platforms. It supports numerous providers, including DigitalOcean Spaces, Amazon S3, Google Drive, and more. Its functionality is similar to that of rsync, but it is optimized for cloud environments.

Why use Rclone with DigitalOcean Spaces?

Using Rclone with DigitalOcean Spaces offers several benefits:

Efficient CLI file management: Execute file operations straight from your terminal.
Seamless integration: DigitalOcean Spaces is S3-compatible, allowing for straightforward interactions.
File synchronization: Keep local directories and remote Spaces in sync with ease.
Handling large file transfers: Manage large files without hassle.
Built-in encryption and compression: Secure and optimize your data automatically.
Automation-friendly: Easily script and schedule routine file transfers.
Cross-platform support: Available for Windows, macOS, and Linux.
Open-source reliability: Benefit from community support and transparent development.

Installing Rclone

Begin by installing Rclone on your system. The installation commands vary depending on your operating system:

For Linux and macOS:

curl -fsSL https://rclone.org/install.sh | sudo bash

Alternatively, you can use a package manager.

On macOS using Homebrew:

brew install rclone

On Linux (Ubuntu/Debian) using apt:

sudo apt install rclone

For Windows:

Download the latest version from the Rclone downloads page.
Extract the ZIP file.
Move rclone.exe to a directory included in your system's PATH.

Verify your installation with:

rclone version

Configuring Rclone for DigitalOcean Spaces

Since DigitalOcean Spaces is S3-compatible, you can configure Rclone to use the S3 protocol. Follow these steps to set up your remote:

Run the Rclone configuration wizard:
```
rclone config
```
Type n to create a new remote.
Enter a name for your remote (e.g., do-spaces).
At Storage, enter s3. At provider, enter DigitalOcean. Use these identifiers rather than menu numbers, which can change between versions.
Keep env_auth set to false and enter your DigitalOcean Spaces credentials at access_key_id and secret_access_key. Generate these keys in your DigitalOcean control panel.
Leave region and location_constraint blank. Set endpoint to your DigitalOcean Spaces endpoint (e.g., nyc3.digitaloceanspaces.com), which includes the region.
Keep the remaining defaults unless your application needs different settings. For advanced configuration, type n when prompted.
Confirm that your settings are correct.

Your do-spaces remote is now ready for use.

Basic Rclone commands

With your remote configured, you can start managing files in DigitalOcean Spaces via the command line.

Listing files

List all files in a specific Space (bucket):

rclone ls do-spaces:your-bucket-name

List files with additional details:

rclone lsl do-spaces:your-bucket-name

List all buckets (Spaces):

rclone lsd do-spaces:

Remember to replace your-bucket-name with your actual bucket name.

Uploading files

To upload a single file:

rclone copy /path/to/local/file do-spaces:your-bucket-name

To upload an entire directory:

rclone copy /path/to/local/directory do-spaces:your-bucket-name

Downloading files

Download a file from your Space:

rclone copy do-spaces:your-bucket-name/remote/file /path/to/local/destination/

Synchronizing directories

Unlike rclone copy, rclone sync deletes destination files that are absent from the source. Check the direction carefully and review a dry run before allowing deletion. To preview making your Space match a local directory:

rclone sync /path/to/local/directory do-spaces:your-bucket-name --dry-run

To preview making a local directory match your Space, including deleting local-only files:

rclone sync do-spaces:your-bucket-name /path/to/local/directory --dry-run

Remove --dry-run only after reviewing the planned changes and keeping any necessary backups. Use rclone copy instead when destination-only files must be preserved.

Automating tasks with Rclone

Rclone's ability to automate file management tasks is one of its greatest strengths. You can schedule commands using cron on Linux/macOS or Task Scheduler on Windows.

Automating with cron (linux/macos)

Create a shell script (for example, sync_to_spaces.sh):

#!/bin/bash
rclone copy /path/to/local/directory do-spaces:your-bucket-name

Make the script executable:
```
chmod +x sync_to_spaces.sh
```
Edit your crontab using:
```
crontab -e
```
Add the following line to run the script daily at 2 AM:
```
0 2 * * * /path/to/sync_to_spaces.sh
```

Automating with task scheduler (Windows)

Create a batch file (for example, sync_to_spaces.bat):

rclone sync C:\path\to\local\directory do-spaces:your-bucket-name

Open Task Scheduler and create a new task to run the batch file on your chosen schedule.

Advanced Rclone features

Rclone includes several powerful features to further refine your file management workflow.

Bandwidth limiting

Prevent Rclone from consuming all your available bandwidth by limiting the upload or download speed. For example, to limit the upload speed to 1MB/s:

rclone copy --bwlimit 1M /path/to/local/file do-spaces:your-bucket-name

Excluding files

Exclude specific files or patterns during synchronization. For example:

rclone sync /path/to/local/directory do-spaces:your-bucket-name --exclude "*.tmp" --exclude "cache/**"

Logging

Enable detailed logging to troubleshoot or monitor operations:

rclone sync /path/to/local/directory do-spaces:your-bucket-name --log-file /path/to/logfile.log --log-level INFO

Encryption

Ensure your files are encrypted before uploading for added security. To set up an encrypted remote:

Run the configuration wizard:
```
rclone config
```
Create a new remote and assign it a name (e.g., do-spaces-encrypted).
At Storage, enter crypt for Encrypt/Decrypt a remote.
At remote (Remote to encrypt/decrypt), enter do-spaces:your-bucket-name.
Configure the encryption options as desired.

Once configured, using the remote do-spaces-encrypted will automatically encrypt your files before uploading.

Troubleshooting common issues

Connection errors

If you experience connection issues:

Verify your internet connection is stable.
Double-check your Rclone configuration details, including the correct endpoint and region.
Ensure your Access Key and Secret Key are valid.

Permission denied errors

If you encounter permission issues:

Confirm that your DigitalOcean Spaces Access Key has the necessary permissions.
Ensure the targeted Space exists and that you are using the correct bucket name.

Sync conflicts

File modifications on either side might lead to conflicts. To prevent data loss:

Use the --backup-dir flag to save overwritten files:

rclone sync /path/to/local/directory do-spaces:your-bucket-name --backup-dir do-spaces:your-bucket-name-backup

Alternatively, use the --checksum flag to compare files based on their checksums instead of modification dates.

Access issues

If you have trouble accessing files after uploading:

Check and adjust the file permissions in DigitalOcean Spaces.
Verify that the files are placed in the correct directory or path.

Conclusion

Rclone is an effective, open-source tool for managing files in DigitalOcean Spaces via the command line. Whether you need to upload, download, synchronize, or automate file management, Rclone simplifies your workflow while handling large files and routine tasks with ease.

By integrating Rclone into your workflow, you can enhance productivity and streamline file operations. For more advanced file processing and transformation workflows, consider exploring Transloadit's File Exporting service.