#post-mortem

Here are our posts tagged post-mortem:

• Fixing Amazon S3 Bugs

  Amazon S3 is great. Except when it isn't. When you are using it a lot - we have stored around one million customer objects so far - you begin to see some odd things that look more...
  Felix Geisendörfer  ·  May 16, 2011

• Last Night's Elevated Queue Times

  Last night we experienced an outage that we promised to shed some more light on. At 22.35 CET it came to our attention that we had elevated queue times.
  Kevin van Zonneveld  ·  March 31, 2012

• Post Mortem: jQuery SDK Problems

  Unfortunately, we've been having a few problems with our jQuery SDK these past two days. We are very sorry about that and sadly, we don't really have a better explanation to give o...
  Tim Koschützki  ·  October 23, 2014

• Post Mortem: S3 Saving Incident

  Yesterday afternoon, we unfortunately had an incident that lasted for approximately 90 minutes. This incident was caused by a bug that we placed into the system ourselves.
  Tim Koschützki  ·  February 17, 2015

• We Failed to Provide the Support You Deserve

  We are writing this to let you know that something went terribly wrong at Transloadit.
  Kevin van Zonneveld  ·  September 25, 2015

• Rate limiter outage post mortem

  Yesterday at 10:49 a.m., we had an issue that affected 2.7% of our customers.
  Tim Koschützki  ·  November 27, 2015

• A Word on ImageTragick

  Early this morning, word came out about a critical bug in ImageMagick, which was quickly nicknamed ImageTragick. Transloadit is a sponsor and long time user of ImageMagick, so it i...
  Kevin van Zonneveld  ·  May 4, 2016

• Redis problems

  Sometimes, things go wrong at Transloadit. When that happens, we want to be transparent about it and own up to our mistakes. In that light, I'm disclosing an issue that affected 81...
  Kevin van Zonneveld  ·  May 30, 2016

• We found and fixed a bug in our data usage calculation

  We recently encountered a bug in our data usage calculation which has affected the invoices of 25% of our customers for the month of June. As a result of this bug, between June 15...
  Tim Koschützki  ·  July 9, 2017

• Accidental breaking change for the crop resize strategy

  On 4 June at 5:32pm CEST, we deployed a backward-compatibility-breaking change, even though we pledge to never 'just do' this and, for example, take extreme care in stack herding t...
  Kevin van Zonneveld  ·  June 6, 2018

• Ghostscript security incident

  On August 21, 2018, Tavis Ormandy of Google's Project Zero disclosed security issues with the way that ImageMagick delegates work to Ghostscript.
  Kevin van Zonneveld  ·  August 30, 2018

• Post-mortem of the outage on December 6

  On December 6, the thing we always dread the most happened. At 13:23, we noticed capacity issues with our platform. It soon turned out to be a serious issue which resulted in downt...
  Kevin van Zonneveld  ·  December 19, 2018

• Vulnerability disclosure

  On Thursday June 20, 2019, our support captain raised a red alert when a vulnerability was reported that a Transloadit server could be rooted. This is the maximum amount of privile...
  Kevin van Zonneveld  ·  July 30, 2019