Export files to MinIO: a complete cURL guide
Exporting files to MinIO using cURL provides a flexible and powerful way to manage object storage operations from the command line. MinIO is a high-performance, S3-compatible object storage system that enables efficient file management and storage solutions. This guide demonstrates how to use cURL with MinIO effectively, covering installation, authentication, and automation techniques.
Introduction to MinIO and its purpose
MinIO is an open-source object storage system that provides high performance and S3 compatibility. It is designed for large-scale data infrastructure and cloud-native applications. MinIO supports features like encryption, identity management, and multi-site replication, making it suitable for enterprise deployments.
Overview of cURL for file transfers
cURL is a versatile command-line tool for transferring data using various protocols. It supports HTTP, HTTPS, and FTP; MinIO’s S3-compatible API uses HTTP or HTTPS. Its extensive features and wide platform support make it a reliable choice for file operations.
Setting up your environment: MinIO and cURL
Prerequisites
Use a running MinIO deployment and an existing bucket. Install the
MinIO Client as mc on your PATH, curl, and Node.js 22 or newer.
In your script directory, install the Node.js SDK:
npm install minio@8
Initial setup
Set MINIO_ENDPOINT, MINIO_ACCESS_KEY, and MINIO_SECRET_KEY in your environment, then configure
the client. Use credentials restricted to the required bucket and operations. An endpoint such as
http://localhost:9000 is suitable only for local development; use HTTPS for remote connections.
mc alias set myminio "$MINIO_ENDPOINT" "$MINIO_ACCESS_KEY" "$MINIO_SECRET_KEY"
Securing access to your MinIO bucket
Implement these security measures for safe file operations:
- Enable TLS/SSL for encrypted connections.
- Use environment variables for credentials.
- Implement proper bucket policies.
- Use pre-signed URLs for temporary access.
Exporting files to MinIO with cURL
Authentication methods
MinIO supports three main authentication approaches:
- MinIO Client (mc) for direct operations
- Pre-signed URLs for temporary access
- Official SDKs for programmatic access
Generate pre-signed URLs
mc share upload generates a POST-policy upload command, not a URL for a PUT request. Do not extract
its URL and reuse it with curl --upload-file. For a presigned PUT, use the
SDK's presignedPutObject.
Save this CommonJS script as presign.cjs:
const Minio = require('minio')
async function main() {
const [bucket, objectName] = process.argv.slice(2)
const { MINIO_ENDPOINT, MINIO_ACCESS_KEY, MINIO_SECRET_KEY } = process.env
if (!bucket || !objectName || !MINIO_ENDPOINT || !MINIO_ACCESS_KEY || !MINIO_SECRET_KEY) {
throw new Error('Provide a bucket, object name, endpoint, and credentials')
}
const endpoint = new URL(MINIO_ENDPOINT)
if (!['http:', 'https:'].includes(endpoint.protocol)) {
throw new Error('The endpoint must use HTTP or HTTPS')
}
const client = new Minio.Client({
endPoint: endpoint.hostname,
port: Number(endpoint.port || (endpoint.protocol === 'https:' ? 443 : 80)),
useSSL: endpoint.protocol === 'https:',
accessKey: MINIO_ACCESS_KEY,
secretKey: MINIO_SECRET_KEY,
})
console.log(await client.presignedPutObject(bucket, objectName, 7200))
}
main().catch(() => {
console.error('Unable to generate an upload URL; check configuration and bucket access')
process.exitCode = 1
})
The URL is a bearer credential valid for two hours. Do not publish it or include it in logs. Run this script on a trusted machine; never distribute the access and secret keys to browser clients.
Automation example
#!/bin/bash
set -euo pipefail
BUCKET="mybucket"
upload_file() {
local file_path="$1"
local file_name
local presigned_url
file_name=$(basename "$file_path")
echo "Generating pre-signed URL for ${file_name}..."
presigned_url=$(node presign.cjs "$BUCKET" "$file_name") || return 1
echo "Uploading ${file_name}..."
if curl -X PUT \
--upload-file "${file_path}" \
--fail-with-body \
--silent \
--show-error \
"${presigned_url}"; then
echo "Uploaded ${file_name}"
return 0
else
echo "Failed to upload ${file_name}" >&2
return 1
fi
}
upload_file "${1:?Usage: bash upload.sh /path/to/file}"
Advanced techniques: handling complex file structures
Large file uploads
For large files, prefer mc cp, which manages multipart uploads automatically rather than sending
one presigned PUT. The supported object size depends on the server configuration:
mc cp large-file.zip myminio/mybucket/large-file.zip
Directory synchronization
Keep local and MinIO directories in sync:
mc mirror local/directory myminio/mybucket/
Common issues & solutions
-
Connection refused:
- Verify that the MinIO server is running.
- Check firewall settings.
- Confirm the correct endpoint and port are used.
-
Access denied:
- Verify credentials.
- Check the bucket policy.
- Ensure proper permissions are set.
-
SSL/TLS errors:
- Use the correct protocol (http or https).
- Verify the certificate if using SSL.
- Set the useSSL option correctly in the SDK.
Monitoring and logging
Track upload progress and log only the HTTP status. This Bash example preserves failure through the logging pipeline:
set -euo pipefail
presigned_url=$(node presign.cjs mybucket large-file.zip)
curl --fail-with-body \
--upload-file "large-file.zip" \
--progress-bar \
--write-out "%{http_code}\n" \
--output /dev/null \
"${presigned_url}" | tee -a upload.log
Using Transloadit for optimized MinIO exports
For advanced file processing and exports to MinIO, consider using Transloadit. Transloadit provides robust file processing capabilities and seamless integration with various storage solutions, including MinIO.
Conclusion
This guide covered essential aspects of using cURL with MinIO for file exports—from installation and secure access to automation and handling complex file structures. The provided scripts and examples demonstrate secure and efficient file operations. For more sophisticated file processing needs, explore Transloadit's comprehensive file handling solutions at transloadit.com.