We're excited to share that tus, our open standard for resumable file uploads, was featured in this month's edition of the Dutch Linux Magazine. For our English speaking readers, here's a translation of the article.
tus - a new open internet protocol
It has probably happened to you: you are trying to upload a fairly large file, but suddenly your connection drops and the upload is aborted. After the connection is restored, you have to start from the beginning again. Really annoying. In many places in the world, the situation is so dire that the connection drops every five minutes. No matter how often you try again, the receiver only gets the first part every time.
In 2009, I started Transloadit together with two guys from Berlin. It is an online service for uploading and encoding files. We noticed an increase in complaints about failing uploads, while our own platform was stable. It was a result of video files growing increasingly larger and the quality of network connections lagging behind. What it comes down to is that we are trying to cross the same bridges, but carrying ten times as much baggage. Tunnels, basements, bars, sparsely populated areas, greedy telecom providers and Wi-Fi interference, they caused plenty of obstacles.
Since we couldn’t do much about the things that cause this problem, we began to look for a solution that would be better suited to withstand them: ‘tus’. By using this, we were able to resume an upload at the point where it was aborted, as soon as the connection was restored. If we tried twenty times, but succeed in sending the next 5% each time, then the entire file would be transferred. The user wouldn’t even have to notice the interruptions, because they were ‘resumable’ uploads.
Downloads have already been resumable ever since HTTP version 1.1. For uploads, however, there was no open standard. There have been a number of projects, but those lacked a common language. They often had a commercial aim or were focused on their own situation. It is like as if there was no NFL and every football team played by their own rules.
We are only a small company. By making the protocol open source, we received cooperation from developers at, among others, Google, Node.js, HTTP/1.1 and ZeroMQ. We are also very proud that Vimeo will be using ‘tus’. Not only for uploads to their platform, but also for file transfer between their servers because of the increased speed that ‘tus’ offers. Vimeo has also provided us with a developer to assist us with building ‘tus’ implementations in more languages. This great degree of cooperation has made ‘tus’ stronger. It is important for the protocol that many people implement it, even our competitors.
We believe this will make the internet better for everyone. We hope that enough companies that prefer to outsource high-quality processing of their uploads will remain, and that they then opt for the source, Transloadit. And if not, then our existing customers at least have the best solution.
(S)FTP and BitTorrent
There is some overlap between ‘tus’ and other protocols. ‘tus’ is, however, based on HTTP and therefore compatible with existing browsers and servers, without the need for large libraries or additional firewall permissions. As a result of this, it can also easily be combined with SSL, load balancers and proxies. ‘tus’ is a document that contains rules about how resumable uploads should work. It can only be of any value when developers abide by these rules in their software. Only then will a ‘language’ emerge that is commonly spoken by the machines and everyone will then have reliable uploads. We are helping this along a bit by making implementations in all of the prominent programming languages available for free. ‘tus’ is also small at its core and offers optional extensions, such as ‘concat’, which, similar to BitTorrent, is able to break up files in smaller pieces and then send those simultaneously. That way, full use is made of a connection and uploads will finish sooner, and we can work around maximum-upload-size limits that some systems pose.
‘tus’ is ready for production. Have a look on http://tus.io, we probably already have an implementation available for your favorite platform. And if that is not the case, we could certainly use your help! Feel free to contact us if you would like to contribute to ‘tus’.
Dutch readers can pick up the magazine in kiosks as of today to read all about tus in their native tongue on page 25, as well as other exciting news from the open source community.