Exporting files to MinIO using cURL provides a flexible and powerful way to manage object storage operations from the command line. MinIO is a high-performance, S3-compatible object storage system that enables efficient file management and storage solutions. This guide demonstrates how to use cURL with MinIO effectively, covering installation, authentication, and automation techniques.

Introduction to MinIO and its purpose

MinIO is an open-source object storage system that provides high performance and S3 compatibility. It is designed for large-scale data infrastructure and cloud-native applications. MinIO supports features like encryption, identity management, and multi-site replication, making it suitable for enterprise deployments.

Overview of cURL for file transfers

cURL is a versatile command-line tool for transferring data using various protocols. It supports HTTP, HTTPS, FTP, and S3, making it ideal for interacting with MinIO’s S3-compatible API. Its extensive features and wide platform support make it a reliable choice for file operations.

Setting up your environment: MinIO and cURL

Prerequisites

Install MinIO Server:

wget https://dl.min.io/server/minio/release/linux-amd64/minio
chmod +x minio

Install MinIO Client:

wget https://dl.min.io/client/mc/release/linux-amd64/mc
chmod +x mc

Version compatibility

Tested with:

  • MinIO Server RELEASE.2024-02-09T21-25-16Z
  • MinIO Client mc RELEASE.2024-02-09T22-18-24Z
  • Node.js SDK 7.1.3

Initial setup

Configure the MinIO client:

mc alias set myminio http://localhost:9000 YOUR-ACCESS YOUR-SECRET

Securing access to your MinIO bucket

Implement these security measures for safe file operations:

  1. Enable TLS/SSL for encrypted connections.
  2. Use environment variables for credentials.
  3. Implement proper bucket policies.
  4. Use pre-signed URLs for temporary access.

Exporting files to MinIO with cURL

Authentication methods

MinIO supports three main authentication approaches:

  1. MinIO Client (mc) for direct operations
  2. Pre-signed URLs for temporary access
  3. Official SDKs for programmatic access

Generate pre-signed URLs

Using MinIO Client:

mc alias set myminio http://localhost:9000 YOUR-ACCESS YOUR-SECRET
mc share upload myminio/mybucket/file.txt --expire 2h

Using Node.js SDK:

const Minio = require('minio')
const client = new Minio.Client({
  endPoint: 'localhost',
  port: 9000,
  useSSL: false,
  accessKey: 'YOUR-ACCESS',
  secretKey: 'YOUR-SECRET',
})

await client.presignedPutObject('mybucket', 'file.txt', 7200)

Automation example

#!/bin/bash
set -euo pipefail

MINIO_ALIAS="myminio"
BUCKET="mybucket"
EXPIRY="2h"

upload_file() {
    local file_path="$1"
    local file_name=$(basename "$file_path")

    echo "Generating pre-signed URL for ${file_name}..."
    presigned_url=$(mc share upload \
        "${MINIO_ALIAS}/${BUCKET}/${file_name}" \
        --expire "${EXPIRY}" --json | jq -r .url)

    echo "Uploading ${file_name}..."
    if curl -X PUT \
        --upload-file "${file_path}" \
        --fail \
        --silent \
        --show-error \
        "${presigned_url}"; then
        echo "✓ Uploaded ${file_name}"
        return 0
    else
        echo "✗ Failed to upload ${file_name}"
        return 1
    fi
}

# Setup (run once)
mc alias set "${MINIO_ALIAS}" \
    "http://localhost:9000" \
    "YOUR-ACCESS" \
    "YOUR-SECRET"

Advanced techniques: handling complex file structures

Large file uploads

For files larger than 5GB, use multipart uploads:

mc cp --recursive local/directory myminio/mybucket/

Directory synchronization

Keep local and MinIO directories in sync:

mc mirror local/directory myminio/mybucket/

Common issues & solutions

  1. Connection refused:

    • Verify that the MinIO server is running.
    • Check firewall settings.
    • Confirm the correct endpoint and port are used.

  2. Access denied:

    • Verify credentials.
    • Check the bucket policy.
    • Ensure proper permissions are set.

  3. SSL/TLS errors:

    • Use the correct protocol (http or https).
    • Verify the certificate if using SSL.
    • Set the useSSL option correctly in the SDK.

Monitoring and logging

Track upload progress and maintain logs with the following command:

curl -X PUT \
  --upload-file "large-file.zip" \
  --progress-bar \
  --write-out "%{http_code}\n" \
  --output /dev/null \
  "${presigned_url}" | tee -a upload.log

Using Transloadit for optimized MinIO exports

For advanced file processing and exports to MinIO, consider using Transloadit. Transloadit provides robust file processing capabilities and seamless integration with various storage solutions, including MinIO.

Conclusion

This guide covered essential aspects of using cURL with MinIO for file exports—from installation and secure access to automation and handling complex file structures. The provided scripts and examples demonstrate secure and efficient file operations. For more sophisticated file processing needs, explore Transloadit's comprehensive file handling solutions at transloadit.com.