Import files from Amazon S3 in Java
To import an Amazon S3 object into a local file, use AWS SDK for Java v2’s getObject with a
file response transformer. The importer below downloads to a temporary location, checks a trusted
SHA-256 checksum, and publishes the destination only after the complete download passes. It also
supports the standard asynchronous client, the AWS CRT client, and S3 Transfer Manager.
Introduction to Amazon S3 and Java
This example retrieves one known key from a general purpose S3 bucket. You need the bucket’s region, permission to read that object, and a SHA-256 supplied by its publisher or calculated from the original file before upload. Keep the object unchanged during the download. The checksum identifies the bytes you intend to import, including an empty object or a binary file.
Setting up AWS SDK v2
The commands use Bash on Linux, the C.UTF-8 locale, and a separately installed JDK and Maven.
The UTF-8 locale keeps non-ASCII key and filename arguments intact. The tested versions are
OpenJDK 21.0.12.1, Maven 3.10.0, AWS SDK v2 2.55.13, and
AWS CRT 0.48.5. The dependency versions below are reproducible test pins; review
AWS SDK releases when updating them.
Create a fresh project outside a Maven checkout that has a .mvn directory. Maven can inherit
that directory’s JVM options, extensions, and command options. This paste checks the tools and
refuses that inherited configuration before creating files; it leaves your shell in its current
directory. If s3-import already exists, use another new directory instead of deleting its files.
(
for tool in java javac mvn; do
command -v "$tool" >/dev/null || {
printf 'Missing prerequisite: %s\n' "$tool" >&2
exit 1
}
done
parent="$PWD"
while :; do
[ ! -e "$parent/.mvn" ] || {
printf 'Use a directory outside the enclosing Maven configuration.\n' >&2
exit 1
}
[ "$parent" != / ] || break
parent="${parent%/*}"
[ -n "$parent" ] || parent=/
done
mkdir s3-import && mkdir -p s3-import/src/main/java
)
Save this complete pom.xml in s3-import/. The BOM keeps the SDK modules at the same version;
CRT has its own version. The dependency plugin copies the runtime JARs needed by the Java launcher.
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>example</groupId>
<artifactId>s3-import</artifactId>
<version>1.0.0</version>
<properties>
<maven.compiler.release>21</maven.compiler.release>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>bom</artifactId>
<version>2.55.13</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
<dependencies>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>s3</artifactId>
</dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>s3-transfer-manager</artifactId>
</dependency>
<dependency>
<groupId>software.amazon.awssdk.crt</groupId>
<artifactId>aws-crt</artifactId>
<version>0.48.5</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-nop</artifactId>
<version>1.7.36</version>
<scope>runtime</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.14.1</version>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<version>3.8.1</version>
<executions>
<execution>
<phase>package</phase>
<goals><goal>copy-dependencies</goal></goals>
<configuration><includeScope>runtime</includeScope></configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
Configuring AWS credentials
Configure credentials through your organization’s existing AWS login or runtime role. This
program uses the SDK’s default credentials provider chain,
which checks Java system properties, environment variables, shared profiles, and runtime roles.
Temporary environment credentials require all three values: AWS_ACCESS_KEY_ID,
AWS_SECRET_ACCESS_KEY, and AWS_SESSION_TOKEN. Keep them out of Java source and Maven files.
The example’s dependencies support environment credentials and basic shared profiles. A profile
that obtains credentials through SSO or AssumeRole also needs the matching SDK modules, such as
sso, ssooidc, or sts, at the BOM version. Complete that login before running the importer.
Configuring the AWS region
Pass the bucket’s region as the second program argument, such as us-east-1. Each client uses
Region.of(regionName), so it does not depend on a region in your shell or shared profile. The key
argument is the literal S3 key, not an s3:// URL; do not URL-encode spaces or percent signs yourself.
Importing files from S3
Save this as s3-import/src/main/java/S3FileImporter.java:
import java.io.IOException;
import java.io.InputStream;
import java.nio.file.FileAlreadyExistsException;
import java.nio.file.Files;
import java.nio.file.LinkOption;
import java.nio.file.Path;
import java.security.MessageDigest;
import java.util.HexFormat;
import java.util.Set;
import java.util.concurrent.CompletionException;
import software.amazon.awssdk.core.async.AsyncResponseTransformer;
import software.amazon.awssdk.core.sync.ResponseTransformer;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.S3AsyncClient;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.model.GetObjectRequest;
import software.amazon.awssdk.services.s3.model.S3Exception;
import software.amazon.awssdk.transfer.s3.S3TransferManager;
import software.amazon.awssdk.transfer.s3.model.DownloadFileRequest;
public class S3FileImporter {
public static void main(String[] args) {
try {
run(args);
} catch (Exception error) {
Throwable cause = error;
while (cause instanceof CompletionException && cause.getCause() != null) {
cause = cause.getCause();
}
if (cause instanceof S3Exception s3Error) {
System.err.println("S3 download failed (HTTP " + s3Error.statusCode() + ").");
} else if (cause instanceof IOException || cause instanceof IllegalArgumentException) {
System.err.println(cause.getMessage());
} else {
System.err.println("Download failed; check credentials, connectivity, and local disk space.");
}
System.exit(1);
}
}
private static void run(String[] args) throws Exception {
if (args.length != 6 || !Set.of("sync", "async", "crt", "transfer").contains(args[0])) {
throw new IllegalArgumentException(
"Usage: S3FileImporter <sync|async|crt|transfer> <region> <bucket> <key> <destination> <sha256>");
}
String expectedHash = args[5];
if (!expectedHash.matches("[0-9a-fA-F]{64}")) {
throw new IllegalArgumentException("Supply the original object's 64-digit SHA-256.");
}
Path destination = Path.of(args[4]).toAbsolutePath();
if (Files.exists(destination, LinkOption.NOFOLLOW_LINKS)) {
throw new FileAlreadyExistsException("Destination already exists: " + destination);
}
GetObjectRequest request = GetObjectRequest.builder().bucket(args[2]).key(args[3]).build();
Region region = Region.of(args[1]);
Path stagingDirectory = Files.createTempDirectory(destination.getParent(), ".s3-import-");
Path stagedFile = stagingDirectory.resolve("object.bin");
try {
download(args[0], region, request, stagedFile);
if (!sha256(stagedFile).equalsIgnoreCase(expectedHash)) {
throw new IOException("SHA-256 mismatch; destination was not published.");
}
Files.move(stagedFile, destination);
System.out.println("Downloaded " + Files.size(destination) + " bytes to " + destination);
} finally {
Files.deleteIfExists(stagedFile);
Files.delete(stagingDirectory);
}
}
private static void download(String mode, Region region, GetObjectRequest request, Path file) {
if (mode.equals("sync")) {
try (S3Client client = S3Client.builder().region(region)
.overrideConfiguration(c -> c.retryStrategy(r -> r.maxAttempts(1))).build()) {
client.getObject(request, ResponseTransformer.toFile(file));
}
return;
}
if (mode.equals("async")) {
try (S3AsyncClient client = S3AsyncClient.builder().region(region)
.overrideConfiguration(c -> c.retryStrategy(r -> r.maxAttempts(1))).build()) {
client.getObject(request, AsyncResponseTransformer.toFile(file)).join();
}
return;
}
try (S3AsyncClient client = S3AsyncClient.crtBuilder().region(region)
.minimumPartSizeInBytes(8L * 1024 * 1024).maxConcurrency(2)
.retryConfiguration(r -> r.numRetries(0)).build()) {
if (mode.equals("crt")) {
client.getObject(request, AsyncResponseTransformer.toFile(file)).join();
} else {
try (S3TransferManager manager = S3TransferManager.builder().s3Client(client).build()) {
manager.downloadFile(DownloadFileRequest.builder()
.getObjectRequest(request).destination(file).build()).completionFuture().join();
}
}
}
}
private static String sha256(Path file) throws Exception {
MessageDigest digest = MessageDigest.getInstance("SHA-256");
try (InputStream input = Files.newInputStream(file)) {
byte[] buffer = new byte[8192];
int count;
while ((count = input.read(buffer)) != -1) {
digest.update(buffer, 0, count);
}
}
return HexFormat.of().formatHex(digest.digest());
}
}
From inside s3-import/, build and run the synchronous path. Replace the bucket, key, region,
and checksum with your object’s values. The destination’s parent directory must exist, and the
destination itself must be new. The quoted classpath keeps the wildcard for Java to expand.
mvn -B package &&
LC_ALL=C.UTF-8 java -cp 'target/classes:target/dependency/*' S3FileImporter \
sync us-east-1 your-bucket-name 'path/to/your/file.bin' \
'./downloaded-file.bin' 'REPLACE_WITH_ORIGINAL_SHA256'
A successful command prints the downloaded byte count and destination and exits with status zero.
The checksum check runs after the SDK completes the file and before Files.move publishes it.
It reads the complete file, so existence, a nonempty prefix, or a completed future is insufficient.
Do not obtain the expected checksum by hashing the downloaded file itself.
Handling exceptions
The command exits with status one for a refused destination, bad checksum, SDK failure, or local I/O error. A caught download or checksum failure removes the temporary file and leaves no new final destination. An existing destination is preserved. Run downloads sequentially for a given destination; this example does not coordinate concurrent publishers.
For a missing key, S3 returns 404 when the caller also has s3:ListBucket, and may return 403
without it. A 403 therefore does not distinguish a nonexistent key from denied access. Check the
exact key and permissions using the GetObject permission rules.
For a checksum mismatch, confirm the publisher’s hash and that the object has not changed before
retrying with a new destination. For a credentials or network failure, correct that prerequisite
and repeat the command. Do not treat a printed exception as a successful import.
S3 client types
Choose the mode by changing only the first program argument and using a new destination.
Synchronous client (S3Client)
sync blocks inside getObject until the file response transformer completes. This is a useful
starting point for a command that imports one object.
Asynchronous client (S3AsyncClient)
async uses the standard Java asynchronous client. In an application, its future lets other work
continue while the download runs. This command calls join() before checking the file, closing the
client, or exiting; otherwise the process could finish before the import does.
Enhanced performance client (AWS CRT)
crt uses the AWS CRT-based S3 client,
an implementation of S3AsyncClient that can fetch large objects in byte ranges. Its native
dependency is included in the Maven build. It has fewer configuration options than the standard
client. Measure your workload before choosing it for performance; these commands do not promise a
particular transfer rate.
Optimizing performance
Streaming to disk
All four modes use file destinations instead of loading the entire object into a Java byte array.
The SHA-256 pass also reads in fixed-size buffers. Allow disk space for the whole object and use a
writable destination directory. If the process is forcibly terminated, inspect and remove its
leftover .s3-import-* directory before retrying; the Java finally block cannot clean up a killed
process.
Multipart downloads
transfer uses S3 Transfer Manager
with the explicitly supplied CRT client. Large downloads can use parallel byte-range requests;
the example sets an 8 MiB minimum part size and a maximum of two connections. These are download
ranges, not S3 multipart uploads. The manager’s completion future is awaited before checksum
verification, and the manager is closed before its client.
Retries are disabled in this demonstration so a failed request is reported to the caller. In an application, choose a bounded retry policy and timeouts for the selected client rather than assuming the standard and CRT builders accept identical settings. A download can otherwise wait on a slow connection; this program does not set a whole-transfer deadline.
IAM permissions
For the current version of a known key in a general purpose bucket, grant s3:GetObject on the
object ARN. Listing the bucket is unnecessary for this importer. For example, have your AWS
administrator adapt this policy to the object you need:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::your-bucket-name/path/to/your/file.bin"
}
]
}
This walkthrough assumes an immediately retrievable object encrypted with S3-managed keys. Archived objects may need restoration, and KMS encryption adds key-permission requirements. Consult the S3 user guide’s permission mapping for those cases. This program does not request a particular version ID or handle SSE-C keys.
Best practices
Keep a trusted checksum beside each object you intend to import. If you later integrate this command into a job, pass its nonzero status back to the scheduler and let downstream processing read only the verified final destination. Select immutable keys or add version-aware retrieval when your publisher replaces objects in place.
For importing S3 objects into a managed file-processing workflow, see Transloadit’s S3 Import Robot and Java SDK.