Resumable uploads

When users are uploading files from their device, any network interruption or server issue could cause the upload to fail, usually requiring the entire file to be retransmitted. Resumable uploads can seamlessly recover from these interruptions and provide a more robust, efficient and pleasant user experience.

Transloadit provides two approaches for uploading files to our servers:

  1. Files can be included in the multipart/form-data POST request when creating an Assembly. Any interruption to this request will cause uploads and the Assembly to fail.
  2. Files can be uploaded using the Tus resumable upload protocol. Uploads are recoverable from network or server issues, while also allowing the user to pause and resume the uploads as they wish. Tus is an open and free protocol for resumable file uploads over HTTP with many open-source client implementations for you to use.

This document describes the API behind the second, resumable approach. It consists of two stages, which are described in this document.

Note: Many off-the-shelf integrations, such as the Node SDK or Uppy, use Tus by default under the hood to upload files. If you are using one of these, you don’t need to implement resumable uploads on your own. This documentation is intended for people who either want to develop SDKs, use SDKs without Tus integration, or not use a Transloadit-provided SDK at all.

Stage 1: Create a new Assembly

A new Assembly is created by sending a multipart/form-data POST request to the endpoint for creating Assemblies. With traditional uploads, all files would be included as additional parts in this request. For resumable uploads, the client does not include the files in this request, but only tells the Transloadit API how many files should be uploaded.

This is achieved by adding the num_expected_upload_files field to the multipart POST request. Its value is the number of files that the client wants to upload for this Assembly. Additional fields for controlling the Assembly Instructions, such as params, must also be included.

The following snippet contains an example HTTP request. The client provides the authentication details and Assembly Instructions in the params field. The num_expected_upload_files field specifies that the client wants to upload two files. However, the actual content of these files is not included in this request.

POST /assemblies HTTP/1.1
Host: api2.transloadit.com
Content-Type:multipart/form-data; boundary=----WebKitFormBoundaryIAWBI8vxocZzsG03

------WebKitFormBoundaryIAWBI8vxocZzsG03
Content-Disposition: form-data; name="params"

{"auth":{"key":"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"},"steps":{"encode":{"robot":"/image/resize"}}}
------WebKitFormBoundaryIAWBI8vxocZzsG03
Content-Disposition: form-data; name="num_expected_upload_files"

2
------WebKitFormBoundaryIAWBI8vxocZzsG03--

Assuming that the Assembly creation is successful, the API responds with the corresponding Assembly Status response, which looks like this example snippet:

{
    "ok": "ASSEMBLY_UPLOADING",
    "assembly_id": "b841ea401e1a11e7b37d7bda1b503cdd",
    "assembly_ssl_url": "https://api2-freja.transloadit.com/assemblies/b841ea401e1a11e7b37d7bda1b503cdd",
    "websocket_url": "https://api2-freja.transloadit.com/ws20277",
    "tus_url": "https://api2-freja.transloadit.com/resumable/files/",
    "expected_tus_uploads": 2,
    "started_tus_uploads": 0,
    "finished_tus_uploads": 0,
    ...
}

We can see that the Assembly is in the uploading state and ready to receive uploads. The response includes the assembly_ssl_url property, which uniquely identifies this Assembly. It also includes the tus_url property, defining the endpoint where the files should be uploaded to. The properties expected_tus_uploads, started_tus_uploads, and finished_tus_uploads describe how many files Transloadit expects for this Assembly and how many uploads have been started/finished.

Stage 2: Upload each file

After the Assembly is created in the first stage, the client can now start uploading files to Transloadit's resumable upload server.

Note: Transloadit supports file sizes up to 200 GB. If you require a higher limit for your application, please get in touch.

Transloadit is running a Tus server using the Tusd software. Its URL is provided by the tus_url property in the Assembly Status, as described in the first stage. This Tus upload server adheres to the protocol specification and allows Tus clients to upload files. You can either implement your own Tus client by following the specification or choose one of the open-source client implementations in your programming language.

An upload via Tus proceeds in two steps:

  1. First, an upload resource on the Tus server is created. The client sends a POST request and includes the Assembly URL, the file name and file type. The server responds with an upload URL, where the client can upload the actual file content to.
  2. Upon receiving the upload URL, the client sends a PATCH request to this endpoint with the file content to perform the actual upload. Once the file has been fully transmitted, the Tus server feeds the file seamlessly into your Assembly for processing, without requiring any additional interaction.

More details on the exact semantics behind this interaction can be found in the protocol specification. In the next section, we will focus on the parts that are relevant to the integration with Transloadit.

Upload creation

The first step is to create an upload resource on the Tus server by sending a POST request to the endpoint, as specified by tus_url. Special metadata must be included to associate the upload with the previously created Assembly. In total, there are three values that must be present in the metadata:

  • assembly_url: the Assembly URL obtained from the assembly_ssl_url property in the Assembly Status in the first stage
  • filename: the file's name
  • fieldname: the equivalent to input field names in HTML forms

Any additional metadata will end up as an assembly variable in file.user_meta. You can use this to dynamically do things in your template per file as an alternative to fields, which is shared by all files in an assembly.

In the example request below, we are uploading a file called isaac.png, with a size of 10,000 bytes, to the Assembly with ID 14b1b490447d11e6aba4756b3e9d3a0d and a field name of file-input. The exact details of the metadata encoding using Base64 are described in the protocol specification.

POST /resumable/files/ HTTP/1.1
Content-Length: 0
Host: api2-freja.transloadit.com
Tus-Resumable: 1.0.0
Upload-Length: 10000
Upload-Metadata: assembly_url aHR0cHM6Ly9hcGkyLnRyYW5zbG9hZGl0LmNvbS9hc3NlbWJsaWVzLzE0YjFiNDkwNDQ3ZDExZTZhYmE0NzU2YjNlOWQzYTBk,filename aXNhYWMucG5n,fieldname ZmlsZS1pbnB1dA==

For a correct request, the server creates an upload resource and returns its upload URL in the Location header. For example:

HTTP/1.1 201 Created
Tus-Resumable: 1.0.0
Location: https://api2-freja.transloadit.com/resumable/files/136058f2ef4dc9de3f5c23ceed591545

Data transfer

After the upload's creation, the client has to upload the actual file content to the Tus upload URL, using a PATCH request:

PATCH /resumable/files/136058f2ef4dc9de3f5c23ceed591545 HTTP/1.1
Host: api2-freja.transloadit.com
Tus-Resumable: 1.0.0
Upload-Offset: 0
Content-Length: 10000
Content-Type: application/offset+octet-stream

[content of file]

If one Tus upload is finished, Transloadit will automatically process it using the parameters that you used to create the Assembly – without requiring you to do anything special. Until all Tus uploads have been finished, the Assembly will remain in the ASSEMBLY_UPLOADING state, even if some of the files have been processed already.

These steps are repeated for each file that the client wants to upload. The client can freely choose to upload these files in parallel or in sequence, depending on the application's needs. If you are trying to add more Tus uploads to an Assembly than you specified during the Assembly creation, the additional ones will be silently dropped.

Resuming

If the data transfer fails because the network was interrupted or the user paused the upload, the client can resume the upload from the point where it stopped.

First, the client sends a HEAD request to the upload URL to determine how much data the server was able to receive before the interruption:

HEAD /resumable/files/136058f2ef4dc9de3f5c23ceed591545 HTTP/1.1
Host: api2-freja.transloadit.com
Tus-Resumable: 1.0.0

The response includes the number of received bytes in the Upload-Offset header. For example, the following response shows an upload where 3,000 out of 10,000 bytes have been received:

HTTP/1.1 204 No Content
Tus-Resumable: 1.0.0
Upload-Offset: 3000
Upload-Length: 10000

The remaining 7,000 bytes can then be uploaded using another PATCH request:

PATCH /resumable/files/136058f2ef4dc9de3f5c23ceed591545 HTTP/1.1
Host: api2-freja.transloadit.com
Tus-Resumable: 1.0.0
Upload-Offset: 3000
Content-Length: 7000
Content-Type: application/offset+octet-stream

[remaining content of file]

Additional information about resumable uploads with Tus is available in the Tus FAQ.