Introducing recursive imports for S3 in Transloadit

We are excited to show you a new addition to our /s3/import Robot. Until now, we didn't offer a way to recursively import files from your Amazon S3 buckets to use in Transloadit. Usually, you would need to specify each folder in an Assembly Step — and that can be a lot of work.

At Transloadit, we are always on the lookout for improvements that could make your work easier for you. With recursive imports for S3, it will be much more convenient to import deeply nested folders — or even an entire S3 bucket at once!

How does it work?

You can supply a path parameter to the Robot, just like before. With recursive: false (set to false by default, to not break BC) only those files directly in the folder represented by that path will be imported. No other files contained in subfolders in that path will be included. With recursive: true, those files would be imported as well.

And if you supply path: '/', 'recursive': true, you can now import an entire S3 bucket. Amazing, isn't it?

By default, the Robot imports a thousand files per invocation. However, there is support for pagination with the page_number and files_per_page parameters. If you are importing a large chunk of files and use the Robot with pagination, you should make sure that no files are added to the imported path in the meantime. Otherwise, some files can be omitted by the pagination.

The Template

{
  "steps": {
    "imported": {
      "robot": "/s3/import",
      "result": true,
      "credentials": "s3_cred",
      "path": "/",
      "recursive": true
    },
    "resized": {
      "use": ["imported"],
      "robot": "/image/resize",
      "result": true,
      "height": 130,
      "imagemagick_stack": "v2.0.3",
      "width": 130,
      "zoom": false
    },
    "exported": {
      "use": ["imported", "resized"],
      "robot": "/s3/store",
      "credentials": "s3_cred",
      "path": "${file.original_path}-resized@130/${file.basename}.${file.ext}"
    }
  }
}

The Assembly Instructions above show a typical instruction to import files and resize them using the /s3/import Robot. Note the changes to two key parameters that differentiate these Instructions for an incursive import from Instructions for a normal import:

• The presence of "recursive": true.
• The path parameter passed is the root of the S3 bucket.

When finished, all imported files will appear in the Assembly result JSON.

Pagination support

As said, the /s3/import Robot will process a thousand files at once. But what if you don't need to import those first thousand files, or maybe the tenth batch of a thousand files? To help with that, we have put together support for pagination. You can now also direct the Robot to the exact page it should start importing from. Take a look at the Assembly Instructions below:

{
  "steps": {
    "imported": {
      "robot": "/s3/import",
      "result": true,
      "credentials": "s3_cred",
      "path": "/",
      "recursive": true,
      "page_number": 10,
      "files_per_page": 500
    },
    "resized": {
      "use": ["imported"],
      "robot": "/image/resize",
      "result": true,
      "height": 130,
      "imagemagick_stack": "v2.0.3",
      "width": 130,
      "zoom": false
    },
    "exported": {
      "use": ["imported", "resized"],
      "robot": "/s3/store",
      "credentials": "s3_cred",
      "path": "${file.original_path}-resized@130/${file.basename}.${file.ext}"
    }
  }
}

As you can see, we have told the Robot to start from the tenth page and recursively import five hundred files per page.

There are a couple of things to take special note of here:

• if you supply a page number that is out of index, the Robot will return no results.
• if you supply a files_per_page parameter that is greater than 1000, it will generate a validation error.
• if you use recursive imorts, all files will be imported in alphabetical order.

That's it for us again. Happy importing! And if you have questions, comments or concerns, do let us know.