Today we are excited to announce a big new feature: Realtime encoding.
Realtime encoding allows you to have your videos encoded while they are still uploading. Since our servers can encode video much faster than most of your users can upload it, this means there is literally no more delay between the end of the upload and the video finishing encoding. In the screencast above this makes a 150x speed difference.
Existing customers can enable this feature by simply adding "realtime": true to their video encoding step. We have also updated the docs for it.
If you are new to transloadit, you can get your hands on this by signing up, or trying it out in our demo section.
Technical details
From a technical perspective, realtime encoding works by pumping the video upload stream into ffmpeg via standard input.
While this sounds easy in theory, it is rather difficult to pull off on most stacks. Luckily our system is build on node.js which is excellent for this kind of stream redirection.
To make it happen, we are using our handcrafted file upload parser which writes the uploaded file to disk and gives us progress events along the way. Each progress event feeds a new chunk of the video into ffmpeg. However, if ffmpeg can't keep up, we wait for the standard input stream to drain, this way there can never be any sort of memory build up.
Dealing with errors is also important. If ffmpeg gives us an error during the realtime encoding, we simply abandon it and fall back to the regular encoding.
That's the high level picture of it. If you are interested in more details, just leave a comment in the post. In case you are already heavily invested in your own video stack, we're also available for consulting to bring this technology to your platform. Just contact us.
What's next
This is just one of the steps to bring streaming to all parts of our service. The next thing we need to tackle is streaming realtime encoded videos to amazon s3.
Another challenge are certain "mov" files that have their meta data at the end of the file. Luckily we got Daniel G. Gaytor excited about this, which prompted him to port his python version of qtfaststart to Javascript. So all of this should get sorted out with the next upgrade of our jQuery SDK.
Also, Daniel is currently looking for a remote job, so you should contact him if you are looking to add a really smart hacker to your team.
Final thoughts
Our mission is to improve file uploading on the web, so please let us know what other aspects of it are ruining your day - we'll be happy to build the solution.
--fg
Credits:
The example video uploaded in the screencast was produced by our friends at némata.