Rate limiter outage post mortem
Yesterday at 10:49 a.m., we had an issue that affected 2.7% of our customers.
The problem was that for these customers assemblies were falsely rejected with a
RATE_LIMIT_REACHED error, for a period of two hours.
The issue was caused by a minor improvement (or so we thought) to our rate limiter. Normally, our testsuite should have spotted any invalid behavior before hitting production, but this was an edge case scenario that we did not cover in our tests and the problem thus hit production after all.
Needless to say, we are very sorry about this.
As soon as we were notified about this, we took the following steps to ensure this can never happen again:
We have raised the account based rate limits of all affected accounts to 50,000 as a hotfix to prevent further Assemblies from being rejected.
We have fixed the bug in the rate limiter and deployed the fix within thirty minutes of the report.
We have improved our automated tests, to ensure that invalid behavior cannot pass into production in the future.
We have improved our monitoring of this, so that we will receive an automated phone call as soon as two or more customers get rate limited at the same time. So, in the unlikely event that this ever happens again, we will know it before everybody else does and be able to address it in the early stages.
We have sent a personal email to all affected customers, informing them about this issue. We have also added a 25% discount to the upcoming invoices for all affected customers.
We are very sorry that this happened and we believe to have taken the necessary steps for this to never happen again.
Share your Transloadit project to earn $300
We love seeing how our community uses Transloadit. Tweet @transloadit to get your proposal approved, share what you’ve built in the form of a blog post or a tutorial on your website, and earn a $300 Gift certificate of your choice as well as a full year of the Startup Plan, at no cost after you publish.
Get started for free with the Community Plan, or, if you are a student, get an upgrade via the GitHub Student Developer Pack.
Get started for free
- Free plan with 5 GB encoding credit
- Set monthly spending limits
- Try without a credit card
- Highly available, globally distributed platform
- Cancel at any time
- Costs displayed for each upload/conversion
Follow us on Twitter:Follow @transloadit
We're SO STOKED to announce the Transloadit Community Plan! 🎉🎉🎉 Unlimited uploading, importing and exporting, 5GB of encoding/month, and access to 50 different file conversion features for all.— 🤖 Transloadit (@transloadit) July 2, 2020
Best part? It's free. Forever. 🤑 Find out more https://t.co/zXWLi3Xa0G pic.twitter.com/DlY5xz1mPG
Get started today
Our Community Plan is free forever and suffices for most projects. Signup is instant. No credit card needed.Sign up today