720% S3 upload & Assembly speed boost with gos3cmd
In December 2017, Tim described an internal S3 upload optimization called gos3cmd. It replaced AWS CLI invocations with a small program using the AWS SDK for Go. The aim was to reduce process startup overhead while retaining the command arguments used by our calling code. This was an implementation detail of Transloadit’s infrastructure, not a new customer-facing upload CLI.
The original note attributed a 720% performance improvement to a collection of changes made since August 2017. The accompanying chart shows US-East-1 test Assembly timings from November 20 to December 7, 2017. It does not establish the comparison with August: the note does not document the August baseline, workload, measurement method, or gos3cmd’s individual contribution. The reported improvement is a historical result, not an independently reproducible benchmark or a promise about present-day S3 uploads or Assembly processing.
Tim also mentioned plans to add Go tests before pushing the code, especially for metadata handling. There is no installation command or public release documented here; the note should not be used as instructions for replacing your own AWS CLI installation.
For current command-line access to Transloadit, see the updated Transloadify walkthrough, which uses the official Node.js SDK’s CLI. That CLI creates Assemblies; it is separate from the internal gos3cmd optimization described here.
When evaluating performance for your own workload, measure upload time separately from time spent waiting for processing and exporting results. Keep file sizes, network location, concurrency, and processing Instructions consistent between runs. This historical note supplies no current timings for those measurements.