Effortless file importing from OpenStack Swift in Java

Tim Koschützki

Co-founder · Berlin, Germany ·

OpenStack Swift is a highly scalable object storage system that empowers you to store and retrieve files efficiently in the cloud. In this guide, we explore how to import files from OpenStack Swift into your Java applications using open-source libraries such as OpenStack4j, and we cover best practices for seamless cloud storage integration.

Setting up your Java environment

Before you begin, ensure you have the following prerequisites:

Java Development Kit (JDK) 11 or later
Maven or Gradle for dependency management
OpenStack Swift credentials (authentication URL, username, password, and domain/project name)

Add the OpenStack4j dependency to your project. For Maven, include the following in your pom.xml:

<dependency>
    <groupId>com.github.openstack4j.core</groupId>
    <artifactId>openstack4j</artifactId>
    <version>3.12</version>
</dependency>

For Gradle, add to your build.gradle:

implementation 'com.github.openstack4j.core:openstack4j:3.12'

Connecting to OpenStack Swift

Establish a connection to your OpenStack Swift instance using OpenStack4j. The example below demonstrates project-scoped authentication. Make sure you provide both a domain and a project name to avoid authorization issues:

import org.openstack4j.api.OSClient.OSClientV3;
import org.openstack4j.model.common.Identifier;
import org.openstack4j.openstack.OSFactory;

public class SwiftConnector {
    private final OSClientV3 os;

    public SwiftConnector(String authUrl, String username, String password,
                         String domainName, String projectName) {
        this.os = OSFactory.builderV3()
            .endpoint(authUrl)
            .credentials(username, password, Identifier.byName(domainName))
            .scopeToProject(Identifier.byName(projectName), Identifier.byName(domainName))
            .authenticate();
    }

    public OSClientV3 getClient() {
        return os;
    }
}

Importing files from Swift

The following example shows how to import a file from a Swift container into your Java application. It downloads into a temporary file next to the destination and only moves that file into place once the transfer has completed and the byte count matches Swift's own. Writing straight to localPath would leave a truncated file that looks complete whenever the connection drops.

import org.openstack4j.api.OSClient.OSClientV3;
import org.openstack4j.model.storage.object.SwiftObject;

import java.io.IOException;
import java.io.InputStream;
import java.nio.file.Files;
import java.nio.file.NoSuchFileException;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.nio.file.StandardCopyOption;

public class SwiftFileImporter {
    private final OSClientV3 os;

    public SwiftFileImporter(OSClientV3 os) {
        this.os = os;
    }

    public void importFile(String containerName, String objectName, String localPath)
            throws IOException {
        SwiftObject obj = os.objectStorage()
            .objects()
            .get(containerName, objectName);

        if (obj == null) {
            // NoSuchFileException's single-argument form takes a file, not a sentence.
            throw new NoSuchFileException(
                containerName + "/" + objectName, null, "Object not found");
        }

        Path destination = Paths.get(localPath).toAbsolutePath();
        // The temporary file has to share a directory with the destination, otherwise the move
        // below crosses filesystems and stops being atomic.
        Path partial = Files.createTempFile(destination.getParent(), ".swift-import-", ".part");

        try {
            long written;
            try (InputStream is = obj.download().getInputStream()) {
                written = Files.copy(is, partial, StandardCopyOption.REPLACE_EXISTING);
            }

            // getSizeInBytes() is a primitive long: it is 0 both for an empty object and for a
            // listing that carried no size, so only a positive value is worth checking against.
            long expected = obj.getSizeInBytes();
            if (expected > 0 && written != expected) {
                throw new IOException(
                    "Truncated download of " + objectName + ": expected " + expected
                        + " bytes but wrote " + written);
            }

            Files.move(partial, destination,
                StandardCopyOption.REPLACE_EXISTING, StandardCopyOption.ATOMIC_MOVE);
        } finally {
            Files.deleteIfExists(partial);
        }
    }
}

Handling large files efficiently

For large files, implement efficient streaming with progress tracking to ensure smooth operations without exhausting memory:

import org.openstack4j.api.OSClient.OSClientV3;
import org.openstack4j.model.storage.object.SwiftObject;

import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.nio.file.Files;
import java.nio.file.NoSuchFileException;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.nio.file.StandardCopyOption;

public class LargeFileImporter {
    private static final int BUFFER_SIZE = 1024 * 1024; // 1MB buffer
    private static final long PROGRESS_INTERVAL = 32L * BUFFER_SIZE;
    private final OSClientV3 os;

    public LargeFileImporter(OSClientV3 os) {
        this.os = os;
    }

    public void importLargeFile(String containerName, String objectName, String localPath)
            throws IOException {
        SwiftObject obj = os.objectStorage()
            .objects()
            .get(containerName, objectName);

        if (obj == null) {
            // NoSuchFileException's single-argument form takes a file, not a sentence.
            throw new NoSuchFileException(
                containerName + "/" + objectName, null, "Object not found");
        }

        long contentLength = obj.getSizeInBytes();
        Path destination = Paths.get(localPath).toAbsolutePath();
        Path partial = Files.createTempFile(destination.getParent(), ".swift-import-", ".part");

        try {
            long totalBytesRead = 0;
            try (InputStream is = obj.download().getInputStream();
                 OutputStream out = Files.newOutputStream(partial)) {
                byte[] buffer = new byte[BUFFER_SIZE];
                long nextReport = PROGRESS_INTERVAL;
                int bytesRead;

                while ((bytesRead = is.read(buffer)) != -1) {
                    out.write(buffer, 0, bytesRead);
                    totalBytesRead += bytesRead;

                    // A size of 0 means Swift reported none, and a per-buffer report on a
                    // multi-GB object is thousands of log lines.
                    if (contentLength > 0 && totalBytesRead >= nextReport) {
                        nextReport += PROGRESS_INTERVAL;
                        System.out.printf("Download progress: %.2f%%%n",
                            (double) totalBytesRead / contentLength * 100);
                    }
                }
            }

            if (contentLength > 0 && totalBytesRead != contentLength) {
                throw new IOException(
                    "Truncated download of " + objectName + ": expected " + contentLength
                        + " bytes but wrote " + totalBytesRead);
            }

            Files.move(partial, destination,
                StandardCopyOption.REPLACE_EXISTING, StandardCopyOption.ATOMIC_MOVE);
        } finally {
            Files.deleteIfExists(partial);
        }
    }
}

Implementing error handling and retries

Robust error handling is essential, especially when network issues or transient errors occur. The example below retries the download up to three times before ultimately failing. A missing object is not a transient condition, so it fails immediately rather than burning three attempts and six seconds on a name that will never resolve. Connection failures and retryable HTTP responses are unchecked SDK exceptions, distinct from local I/O failures. Authentication and other permanent HTTP errors are not retried:

import org.openstack4j.api.OSClient.OSClientV3;
import org.openstack4j.api.exceptions.ConnectionException;
import org.openstack4j.api.exceptions.ResponseException;

import java.io.IOException;
import java.nio.file.NoSuchFileException;
import java.util.Set;

public class RetryableSwiftImporter {
    private static final int MAX_RETRIES = 3;
    private static final long RETRY_DELAY_MS = 2000;
    private static final Set<Integer> RETRYABLE_STATUS = Set.of(408, 429, 500, 502, 503, 504);
    private final SwiftFileImporter importer;

    public RetryableSwiftImporter(OSClientV3 os) {
        this.importer = new SwiftFileImporter(os);
    }

    public void importWithRetry(String containerName, String objectName, String localPath)
            throws IOException, InterruptedException {
        Exception lastException = null;

        for (int attempt = 1; attempt <= MAX_RETRIES; attempt++) {
            try {
                // importFile leaves the destination untouched until a whole, size-checked copy has
                // landed, so retrying can never observe a half-written file.
                importer.importFile(containerName, objectName, localPath);
                return;
            } catch (NoSuchFileException e) {
                throw e;
            } catch (ResponseException e) {
                if (!(e instanceof ConnectionException) && !RETRYABLE_STATUS.contains(e.getStatus())) {
                    throw e;
                }
                lastException = e;
            } catch (IOException e) {
                lastException = e;
            }
            if (attempt < MAX_RETRIES) {
                Thread.sleep(RETRY_DELAY_MS * attempt);
            }
        }

        throw new IOException("Failed to import after " + MAX_RETRIES + " attempts", lastException);
    }
}

Error handling considerations

When working with OpenStack Swift, consider the following common error scenarios and tailor your error handling accordingly:

Container Not Found: Ensure the specified container exists in your Swift instance.
Object Not Found: Verify the object name and its path.
Authentication Failures: Double-check your credentials, domain name, and project name for proper scoping.
Quota Exceeded: Monitor your storage limits and usage to avoid surpassing allocated quotas.

Providing detailed error messages and handling these cases explicitly can significantly improve troubleshooting and system reliability.

Performance optimization tips

To optimize your Swift file importing workflow, consider the following practices:

Connection Management:
- Reuse an OSClientV3 within the thread that authenticated it. SwiftObject.download() calls Apis.get(...), which resolves the session OpenStack4j stores in a ThreadLocal, so handing the client or a SwiftObject to a worker thread leaves that worker with no session at all
- For concurrent imports, authenticate once per worker thread, or re-derive a client on the worker from the token you already hold, rather than pooling one shared client
- Set appropriate timeouts based on your use case
Buffer Size Optimization:
- Use larger buffers (1MB or more) for transferring large files
- Adjust the buffer size based on available memory and file sizes
- Monitor memory usage during transfers
Resource Management:
- Always use try-with-resources for proper cleanup of streams and connections
- Download to a temporary file in the destination directory and move it into place only after a size check passes, so a failed transfer never leaves a plausible-looking partial file
- Log exceptions and clean up resources to avoid memory leaks
Security Best Practices:
- Store credentials securely using environment variables or secure vaults
- Use HTTPS endpoints for all connections to protect data in transit
- Implement proper access controls and audit logging
Bulk File Operations:
- When importing numerous files, consider using Swift's native bulk operations if available, or process files sequentially to manage memory more effectively

Conclusion

By leveraging OpenStack4j, Java developers can efficiently integrate OpenStack Swift into their applications for seamless file importing and cloud storage management. The examples and best practices provided here form a robust starting point for building efficient file importing solutions.

For more advanced file handling capabilities and automated processing workflows, consider exploring Transloadit.