WeTransfer stands as a titan in the file-sharing landscape and the company is widely celebrated for its ability to facilitate the effortless transfer of large files. With a staggering 2 petabytes (PB) of file previews generated and served each month, WeTransfer needed a solution that could offer seamless, real-time previews without overburdening its existing infrastructure and dev team—while simultaneously reducing its carbon footprint through innovation. The WeTransfer team discovered the solution in Transloadit's Smart CDN, a service that not only addressed the immediate challenge but also brought efficiency and an unexpected level of adaptability to the table.

A rich blue background with the WeTransfer logo in front.

The challenge: managing a deluge of preview data

Providing instant and reliable file previews for any media type improves the user experience of a file-sharing platform. Martijn Vermaat, Engineering Manager at WeTransfer, underscores the challenge:

Martijn Vermaat

Serving 2 PB of media previews every month is a significant task. It only became more complex with the diverse range of media types we handle. We required a solution that was on-demand, performant, (real-time <600ms latencies), robust, and flexible.

Handling this much data comes with climate considerations, too. Small inefficiencies at the file level ultimately add up to a substantial increase in carbon footprint, something which the WeTransfer team was acutely aware of and wanted to address.

With this in mind, WeTransfer began their search for an appropriate solution that could handle the file-processing demands their service generates.

The solution: Transloadit's Smart CDN and unparalleled support

The WeTransfer team discovered their solution in the Transloadit Smart CDN. It immediately stood out because of its ability to generate file previews on the fly across a wide array of media types through a simple URL request. However, that's not all. Martijn reflects on what truly sets Transloadit apart:

While the Smart CDN’s capabilities were a close match to what we needed, it was Transloadit’s relentless support and preparedness to implement feature requests, as well as its ability to meet evolving targets, that were most astonishing. Their team’s responsiveness and willingness to adapt have played a crucial role in our ongoing success. It is not every day that you see a company in business for so long, still be this agile and customer-focused.

Integration was only the beginning. The WeTransfer team worked closely with us to optimize our encoding platform for their services, ensuring that minimal processing (and climate impact) was needed to handle the enormous amount of data they were sending our way. Some of the things our co-founder Tim looked into, were:

  • Reducing unnecessary work, such as no longer extracting metadata by default for this use case
  • Streaming results directly to the client instead of storing them on S3 as intermediary steps
  • Benchmarking 6 different image resizers under load to establish which had the highest throughput per CPU tick
  • Handling the entire flow on single machines, rather than using the existing cluster that neatly separates concerns by machine roles, but with the trade-off of extra round trips over the network and cache misses
  • Reviewing flame graphs and implementing additional caching to avoid redundant computations

Together, these optimizations resulted in an order of magnitude less compute spend on getting the same end result to the user, saving tremendously on hardware and energy. In addition, these savings translated into bringing our encoding times for WeTransfer’s file previews down from several seconds to 60 ms on average, resulting in a much nicer user experience.

The outcome: enhanced efficiency and a partnership for the future

What began as a humble start together has proven to be a fruitful partnership ongoing since 2021. The team behind WeTransfer faced an exceptional challenge, and it was in part thanks to the Transloadit Smart CDN that they were able to meet it while keeping focus on their core business. Integrating Transloadit's Smart CDN has significantly improved WeTransfer's file preview generation in terms of energy efficiency, user experience, and reliability.

Plans to further reduce carbon footprint together continue, as core engineers from both companies investigate opening a new data center powered by 100% renewable energy sources.

As with any good partnership, there is a strong mutual benefit. Kevin van Zonneveld, co-founder of Transloadit, explains how working together has been advantageous for both companies:

The collaboration with WeTransfer is exemplary of what a mutually beneficial partnership can look like. It has always been a pleasure to interact with WeTransfer's highly talented team members. Their feedback and our expertise have evolved Transloadit's Smart CDN to become best-in-class, which directly benefits all our customers and, by extension, Transloadit. At the same time, it has freed up WeTransfer's infrastructure and team to focus on aspects most crucial to their core mission.

Looking ahead

The collaboration between WeTransfer and Transloadit, built upon a rock-solid foundation in the form of Transloadit’s Smart CDN and exceptional support, sets a new standard for partnership in technology. Martijn concludes:

Our journey with Transloadit has been remarkable. Their support and flexibility have been pivotal in enhancing our service. Having them in our Slack at times felt like an extension of our engineering department. Given Transloadit's focus on continued innovation, expansion into AI, and efforts to bring down carbon emissions, we’re excited about what the future holds and the continued evolution of our partnership.

If you have questions about our Smart CDN or would like to know how Transloadit can be deployed to significantly reduce costs, carbon emissions, time-to-market, and latency, reach out, and talk directly with a human.