Last updated: September 11

Import files from MinIO in Java

Kevin van Zonneveld

Co-founder · Amsterdam, The Netherlands ·

MinIO is a high-performance, distributed object storage system compatible with Amazon S3 APIs. Its simplicity, scalability, and compatibility make it a popular choice for developers looking to integrate robust storage solutions into their Java applications.

In this guide, we'll explore how to efficiently import files from MinIO using Java, leveraging the MinIO Java SDK. We'll cover setup, integration, practical examples, security best practices, and optimization tips.

Introduction to MinIO and S3 compatibility

MinIO provides an open-source, distributed object storage solution designed for high performance and scalability. Its compatibility with Amazon S3 APIs allows developers to easily integrate it into existing workflows and applications that already use S3.

Setting up your Java environment

Before you begin, ensure you have:

Java 8 or later installed.
A running MinIO server instance.
Access credentials (accessKey and secretKey) for your MinIO server.

Integrating the MinIO Java SDK

To integrate the MinIO Java SDK, add the following dependency to your project:

Maven:

<dependency>
  <groupId>io.minio</groupId>
  <artifactId>minio</artifactId>
  <version>8.5.17</version>
</dependency>

Gradle:

implementation 'io.minio:minio:8.5.17'

Importing files from MinIO: a step-by-step guide

Here's a practical example demonstrating how to import a file from MinIO:

import io.minio.MinioClient;
import io.minio.GetObjectArgs;
import io.minio.errors.MinioException;
import java.io.IOException;
import java.io.InputStream;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.security.InvalidKeyException;
import java.security.NoSuchAlgorithmException;

public class MinIOFileImporter {
  public static void main(String[] args) {
    // Initialize and close the client for this command-line operation.
    try (MinioClient minioClient = MinioClient.builder()
        .endpoint("https://play.min.io")
        .credentials(System.getenv("MINIO_ACCESS_KEY"), System.getenv("MINIO_SECRET_KEY"))
        .build()) {

      // Download file from MinIO
      try (InputStream stream = minioClient.getObject(
          GetObjectArgs.builder()
            .bucket("my-bucket")
            .object("path/to/file.txt")
            .build())) {

        Files.copy(stream, Paths.get("downloaded-file.txt"));
        System.out.println("File imported successfully.");
      }
    } catch (MinioException e) {
      System.err.println("MinIO request failed: " + e.getClass().getSimpleName());
    } catch (IOException e) {
      System.err.println("File operation error: " + e);
    } catch (NoSuchAlgorithmException | InvalidKeyException e) {
      System.err.println("Authentication error: " + e);
    } catch (Exception e) {
      System.err.println("Client setup or cleanup failed: " + e.getClass().getSimpleName());
    }
  }
}

Replace the demo endpoint, bucket name, and object path with your MinIO details, and set MINIO_ACCESS_KEY and MINIO_SECRET_KEY in the environment. The demo endpoint uses HTTPS on port 443. Files.copy does not overwrite an existing local file; a failed stream may leave a partial new file, which you should remove or quarantine before retrying.

Handling common exceptions

When working with MinIO, you might encounter several types of exceptions:

MinioException: Base exception for all MinIO-related errors.
- ErrorResponseException: Indicates issues like incorrect bucket names or permissions.
- InsufficientDataException: Occurs when the data stream ends prematurely.
- InternalException: Represents internal SDK errors.
IOException: File system related errors.
NoSuchAlgorithmException or InvalidKeyException: Authentication-related errors.

Always handle exceptions gracefully and log meaningful error messages to simplify debugging.

Security best practices

When working with MinIO in production, follow these security best practices:

Use environment variables

Store credentials in environment variables instead of hardcoding them:

String accessKey = System.getenv("MINIO_ACCESS_KEY");
String secretKey = System.getenv("MINIO_SECRET_KEY");

MinioClient minioClient = MinioClient.builder()
  .endpoint("https://your-minio-server.com")
  .credentials(accessKey, secretKey)
  .build();

Enable TLS

Always use HTTPS in production:

MinioClient minioClient = MinioClient.builder()
  .endpoint("your-minio-server.com", 443, true) // true enables HTTPS
  .credentials(accessKey, secretKey)
  .build();

Implement proper error handling and logging

Log the exception type and operation without dumping HTTP traces or credentials. Within the main example's catch (MinioException e) block, for example:

System.err.println("MinIO download failed: " + e.getClass().getSimpleName());

Connection management and optimization

To optimize MinIO operations, consider these techniques:

Connection pooling

Reuse the MinioClient instance across your application:

Keep it open for the application's lifetime and close it during shutdown.

import io.minio.MinioClient;

// Create a singleton MinioClient
public class MinioClientFactory {
  private static MinioClient instance;

  public static synchronized MinioClient getClient() {
    if (instance == null) {
      instance = MinioClient.builder()
        .endpoint("https://your-minio-server.com")
        .credentials(System.getenv("MINIO_ACCESS_KEY"), System.getenv("MINIO_SECRET_KEY"))
        .build();
    }
    return instance;
  }
}

Configure timeouts

Set appropriate connection and read/write timeouts:

import io.minio.MinioClient;
import okhttp3.OkHttpClient;
import java.util.concurrent.TimeUnit;

OkHttpClient httpClient = new OkHttpClient.Builder()
  .connectTimeout(30, TimeUnit.SECONDS)
  .writeTimeout(60, TimeUnit.SECONDS)
  .readTimeout(60, TimeUnit.SECONDS)
  .build();

MinioClient minioClient = MinioClient.builder()
  .endpoint("https://your-minio-server.com")
  .credentials(System.getenv("MINIO_ACCESS_KEY"), System.getenv("MINIO_SECRET_KEY"))
  .httpClient(httpClient)
  .build();

Implement retry mechanisms

This method retries transient failures while opening a download, using capped exponential backoff. It assumes a MinioClient minioClient field in the surrounding class. Import the types shown below at the top of that file. The caller must close the returned stream and separately handle failures while reading its body:

import io.minio.GetObjectArgs;
import io.minio.errors.ErrorResponseException;
import io.minio.errors.MinioException;
import java.io.IOException;
import java.io.InputStream;
import java.security.InvalidKeyException;
import java.security.NoSuchAlgorithmException;

public InputStream getObjectWithRetry(String bucket, String object, int maxRetries) {
  if (maxRetries < 1) throw new IllegalArgumentException("maxRetries must be at least 1");
  int attempts = 0;
  while (attempts < maxRetries) {
    try {
      return minioClient.getObject(
        GetObjectArgs.builder()
          .bucket(bucket)
          .object(object)
          .build());
    } catch (MinioException | IOException e) {
      if (e instanceof ErrorResponseException) {
        int status = ((ErrorResponseException) e).response().code();
        if (status != 408 && status != 429 && status < 500) {
          throw new RuntimeException("Non-retryable storage response", e);
        }
      } else if (!(e instanceof IOException) && !(e instanceof io.minio.errors.ServerException)) {
        throw new RuntimeException("Non-retryable MinIO failure", e);
      }
      attempts++;
      if (attempts >= maxRetries) throw new RuntimeException("Failed after " + maxRetries + " attempts", e);
      try {
        Thread.sleep(1000L << Math.min(attempts - 1, 5)); // 1s, 2s, 4s, capped at 32s
      } catch (InterruptedException ie) {
        Thread.currentThread().interrupt();
        throw new RuntimeException("Interrupted during retry", ie);
      }
    } catch (NoSuchAlgorithmException | InvalidKeyException e) {
      throw new RuntimeException("Cryptographic configuration error", e);
    }
  }
  throw new RuntimeException("Should not reach here");
}

Real-world use cases

MinIO is widely used for:

Storing and retrieving large datasets for data analytics.
Backup and archival solutions.
Serving static assets for web applications.
Media storage and streaming.
AI/ML model storage and retrieval.

Example: listing objects

import io.minio.ListObjectsArgs;
import io.minio.Result;
import io.minio.messages.Item;

try {
  Iterable<Result<Item>> results = minioClient.listObjects(
    ListObjectsArgs.builder().bucket("my-bucket").prefix("documents/").build());

  for (Result<Item> result : results) {
    Item item = result.get();
    System.out.println(item.objectName());
  }
} catch (Exception e) {
  System.err.println("Error listing objects: " + e.getMessage());
}

Troubleshooting

Connection refused

If you encounter connection issues:

Verify that the MinIO server is running.
Check firewall settings.
Ensure the endpoint URL and port are correct.
Verify network connectivity.

Access denied

For permission issues:

Verify your access and secret keys.
Check bucket policies and permissions.
Ensure the bucket exists.
Verify the object path is correct.

Conclusion

Integrating MinIO with Java provides a powerful and efficient way to manage files in your applications. The MinIO Java SDK offers a robust API for interacting with MinIO servers, including features like connection pooling, retry mechanisms, and comprehensive error handling. By following the best practices outlined in this guide, you can build secure, scalable, and high-performance applications that leverage the power of MinIO.

For further exploration, check out the official MinIO Java SDK documentation.

Transloadit also leverages MinIO. Our 🤖 MinIO Import Robot allows you to import entire directories from your MinIO buckets effortlessly, and our 🔧 Java SDK simplifies integration with Transloadit's file processing capabilities.