We are writing to address a recent issue that has arisen as a result of our change in SSL certificates from GoDaddy to Let's Encrypt. A small number of our customers have reported encountering a CERT_HAS_EXPIRED error when executing Assemblies from older systems. This error message suggests that the client's system believes our server's SSL certificate has expired, preventing a connection.

In light of these issues, we want to provide a brief post-mortem of what occurred, why it happened, and the steps we're taking to prevent such incidents in the future.

  1. What Happened

    On July 3rd, we migrated from GoDaddy to Let's Encrypt for our SSL certificates. On July 4th, two customers started to report a CERT_HAS_EXPIRED error when creating Assemblies.

  2. Root Cause

    After investigating, we realized this issue relates to a known deprecation situation in September 2021 from Let's Encrypt (see https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/ for more information). Essentially, some systems have not updated their trust store (a stored set of trusted certificates) to include the new Let's Encrypt root certificate, and continue to rely on an expired root certificate. This situation may arise particularly in older or less frequently updated production systems. In the two cases reported, it revolved around Docker containers that hadn't been updated for a while (for example: php:7.0-apache). Typically it will be containers, because if the host OS hadn't been updated, it would have had problems connecting to any server that uses Let's Encrypt, which is most of the internet, and so admins would have likely noticed earlier already. It is with purpose built Docker containers only used for connecting to Transloadit, and not being updated for several years, that you could now experience problems.

  3. Why We Missed It

    During our migration and testing process, we failed to consider the potential impact of older systems still relying on the deprecated DST Root CA X3. We deeply regret this oversight.

  4. Steps Moving Forward

    • Communication: In the future, we commit to informing customers well in advance of any changes that could potentially affect your service, giving you the opportunity to prepare and make necessary adjustments.
    • Thorough Testing: We will extend our testing processes to consider the broadest range of potential user situations, including out-of-date systems.
    • Support: We will continue to offer support and solutions for those affected by this change, guiding you through the process of updating your trust stores if necessary.

We recommend those affected to update their system's trust store to include the new Let's Encrypt root certificate, which can usually be done by updating the operating system (of the container) or the relevant software packages. We understand that this may not be a suitable solution for all customers, and we're here to provide guidance and assistance. Please reach out to support if you are affected.

We care a lot about not breaking backwards compatibility at Transloadit and typically offer advisories and graceful upgrade paths across the board. In this case I as founder personally oversaw QA and gradual rollout and deemed the change safe. I was wrong, and I am sorry about that, and the trouble caused.

We will continue to work tirelessly to ensure a secure and reliable service for all of our customers.