Exporting files to Amazon S3 is a common requirement for modern web applications. Whether you're building a file storage system, implementing backups, or managing user uploads, understanding how to efficiently export files to Amazon S3 using Node.js is essential.

Prerequisites

Before we begin, ensure you have:

  • Node.js 20 or later installed
  • An AWS account with access credentials
  • Basic understanding of JavaScript and Node.js
  • A project set up with "type": "module" in your package.json

Setting up AWS credentials

To interact with Amazon S3 from your Node.js application, you'll need to configure your AWS credentials. There are several ways to do this:

  1. Environment Variables: Set the following environment variables in your system or .env file:

    AWS_ACCESS_KEY_ID=YOUR_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY=YOUR_SECRET_ACCESS_KEY
AWS_REGION=YOUR_AWS_REGION

  2. Shared Credentials File: Use the AWS credentials file located at ~/.aws/credentials:

    [default]
aws_access_key_id = YOUR_ACCESS_KEY_ID
aws_secret_access_key = YOUR_SECRET_ACCESS_KEY

  3. IAM Roles: If you're running your application on AWS services like EC2, you can assign an IAM role to your instance and the AWS SDK will automatically use those credentials.

Installing and using the AWS SDK

We'll start with the official AWS SDK for JavaScript (v3). Install the necessary packages with:

npm install @aws-sdk/client-s3 @aws-sdk/lib-storage @aws-sdk/node-http-handler

Here's a basic example of uploading a file to Amazon S3:

import {
  S3Client,
  PutObjectCommand,
  S3ServiceException,
  type PutObjectCommandInput,
} from '@aws-sdk/client-s3'
import { NodeHttpHandler } from '@aws-sdk/node-http-handler'
import { createReadStream } from 'fs'

const s3Client = new S3Client({
  region: 'us-east-1',
  requestHandler: new NodeHttpHandler({
    socketTimeout: 3000,
    timeoutError: true,
  }),
})

async function uploadFile(filePath: string, bucketName: string, key: string) {
  try {
    const uploadParams: PutObjectCommandInput = {
      Bucket: bucketName,
      Key: key,
      Body: createReadStream(filePath),
    }

    const command = new PutObjectCommand(uploadParams)
    await s3Client.send(command)
    console.log('Upload completed successfully')
  } catch (err) {
    if (err instanceof S3ServiceException) {
      console.error(`S3 Error: ${err.name} - ${err.message}`)
    } else {
      console.error('Upload failed:', err)
    }
  }
}

// Usage
await uploadFile('path/to/file.txt', 'my-bucket', 'file.txt')

This example uses the PutObjectCommand to upload a file to S3. The createReadStream function streams the file for efficient uploading.

Handling large files with multipart upload

For large files, it's recommended to use multipart uploads to improve reliability and performance. The AWS SDK provides utilities to simplify this process. Using the Upload class from @aws-sdk/lib-storage handles multipart uploads automatically:

import { S3Client, S3ServiceException } from '@aws-sdk/client-s3'
import { Upload } from '@aws-sdk/lib-storage'
import { createReadStream } from 'fs'

const s3Client = new S3Client({ region: 'us-east-1' })

async function uploadLargeFile(filePath: string, bucketName: string, key: string) {
  try {
    const upload = new Upload({
      client: s3Client,
      params: {
        Bucket: bucketName,
        Key: key,
        Body: createReadStream(filePath),
      },
    })

    upload.on('httpUploadProgress', (progress) => {
      console.log(`Progress: ${progress.loaded}/${progress.total}`)
    })

    await upload.done()
    console.log('Large file upload completed successfully')
  } catch (err) {
    if (err instanceof S3ServiceException) {
      console.error(`S3 Error: ${err.name} - ${err.message}`)
    } else {
      console.error('Upload failed:', err)
    }
  }
}

// Usage
await uploadLargeFile('path/to/large-file.zip', 'my-bucket', 'large-file.zip')

Security best practices

When exporting files to Amazon S3, it's crucial to follow security best practices:

  1. Use IAM Roles: Instead of hardcoding access keys, use IAM roles when deploying to AWS services like EC2 or ECS. The AWS SDK automatically uses HTTPS and loads credentials from environment variables, shared credentials files, or IAM roles if available.

    import { S3Client } from '@aws-sdk/client-s3'
import { NodeHttpHandler } from '@aws-sdk/node-http-handler'

const s3Client = new S3Client({
  region: 'us-east-1',
  maxAttempts: 3,
  requestHandler: new NodeHttpHandler({
    socketTimeout: 3000,
    timeoutError: true,
  }),
})

  2. Implement Least Privilege Access: Restrict your IAM user or role permissions to only those necessary for the task.

    {
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:PutObject"],
      "Resource": "arn:aws:s3:::your-bucket/*"
    }
  ]
}

  3. Enable Server-Side Encryption: Protect your data at rest by enabling server-side encryption.

    const uploadParams: PutObjectCommandInput = {
  Bucket: bucketName,
  Key: key,
  Body: createReadStream(filePath),
  ServerSideEncryption: 'AES256',
}

  4. Use HTTPS: The AWS SDK v3 uses HTTPS by default for all API calls, ensuring data in transit is encrypted.

Error handling and retries

Implement robust error handling and retry logic to make your application resilient. The AWS SDK includes built-in retry logic, but you can customize it as needed:

import {
  S3Client,
  PutObjectCommand,
  S3ServiceException,
  type PutObjectCommandInput,
} from '@aws-sdk/client-s3'

const s3Client = new S3Client({
  region: 'us-east-1',
  maxAttempts: 3,
})

async function uploadFileWithRetry(filePath: string, bucketName: string, key: string) {
  let attempt = 0
  while (attempt < 3) {
    try {
      await uploadFile(filePath, bucketName, key)
      return
    } catch (err) {
      attempt++
      if (attempt === 3) {
        console.error(`Failed to upload after 3 attempts.`)
        throw err
      }
      console.log(`Retrying upload (${attempt}/3)...`)
    }
  }
}

For additional control, you can implement custom retry strategies tailored to your needs.

TypeScript support

The AWS SDK for JavaScript v3 offers robust TypeScript support to help catch errors during development and provide improved autocomplete in your IDE. Below is a basic example demonstrating how to define types when uploading a file:

import { S3Client, PutObjectCommand, type PutObjectCommandInput } from '@aws-sdk/client-s3'
import { createReadStream } from 'fs'

const s3Client = new S3Client({ region: 'us-east-1' })

const uploadParams: PutObjectCommandInput = {
  Bucket: 'my-bucket',
  Key: 'my-key',
  Body: createReadStream('file.txt'),
}

async function uploadFile() {
  try {
    const command = new PutObjectCommand(uploadParams)
    await s3Client.send(command)
    console.log('File uploaded successfully.')
  } catch (error) {
    console.error('Error uploading file:', error)
  }
}

uploadFile()

This example demonstrates how to leverage TypeScript to define and enforce the structure of your parameters, enhancing maintainability and catching potential issues during development.

Conclusion

Efficiently exporting files to Amazon S3 using Node.js involves understanding how to interact with the AWS SDK and following best practices for security and error handling. By leveraging the AWS SDK and implementing the strategies discussed, you can build robust file export functionality in your Node.js applications.

At Transloadit, we focus on making file processing and infrastructure tasks easier. If you're looking for a solution to handle file uploads and processing without the hassle, feel free to check out our services.