At Transloadit, it is our desire to give you the best possible encoding experience, and one of the main factors involved is making sure that we are using the latest encoding tools.

It's no secret that we are heavily relying on open source technology for this, and we therefore make a conscious effort to give back to the giants on whose shoulders we are standing, for instance through providing free hardware to the ImageMagick project.

Even though we have an intimate relationship with the project, it has proven to be very problematic for us to modernize our stack.

The 'fun' parts of maintaining encoding software

Oftentimes, our customers rely on behavior of the old version (even on its bugs, but more on that later), so "just upgrading" - which we naively tried to do a few years back - breaks their workflow. To ensure we can provide a consistent experience even between operating system upgrades, we are required to compile ImageMagick and all of its 31 dependencies. Yes 31, because by adding libraries such as pixman, pango and ufraw, we gain support for 131 more formats than what a standard apt-get install imagemagick would provide on Ubuntu. And by including libraries all the way down to zlib, we also shield ourselves from any backwards compatibility breakage that could occur as a result of operating system upgrades, thereby allowing our encoding capabilities to be the reliable constant upon which our customers can build their businesses.

Besides vast support for formats and this guaranteed continuity, the advantage of compiling is that it allows you to create a static binary. This is a single executable file that has every dependency baked into it. This way, it becomes easy to run very different versions of the same software alongside one another without any conflicts, giving our customers an upgrade path that is opt-in. This is how we have been providing upgrades to our video encoding software.

The problem, however, that we are having now with upgrading our image stack, is that the process of compiling ImageMagick and its 31 dependencies would take two hours, and that is under perfect conditions on a beefy server. If one particular library version conflicted with another, you may only find out after 90 minutes, after which you would have to think of a possible fix and then try all over. That can hardly be considered workable or maintainable, and it certainly does not amount to the level of agility that we aspire to have in our development.

The introduction of Docker containers, we thought, would solve some of this struggle in that we could at least run different versions alongside each other easier, and speed up our build times through the use of layers. Also, instead of having single massive binaries to recompile, we could just recompile a single shared library if needed. Our initial joy was tempered when we learned that there was no way to start containers without dangerous system privileges. This meant we either had to keep containers running and let our non-privileged API user proxy commands to those, or be okay with giving this user the privilege to start containers. Both options weren't great, to say the least. Proxying would be quite fragile, and by handing out such privileges, an attacker could theoretically mount any directory on the host machine as a --volume into the container, and then read/write/delete whatever they wanted. We later also learned about the work that was being done in the runC project to launch containers without root access (and we'll likely take that approach in the future) but it wasn't yet at the level of maturity that we needed.

We also considered running a separate cluster for the new version, but this would have some serious costs in terms of additional hardware, maintenance and operational overhead.

We dove deep into all these different directions, but ended up having to back out every time since there were serious drawbacks. How should the future of our stack maintenance look? What solution is both reliable and mature enough to give us guaranteed continuity while maintaining our agility?

Nix

For a long time, we had also been looking at Nix as a possible solution. We have been building our software with a simple compile framework called depmake that Felix wrote back in 2012 and which already implemented some of Nix' basic ideas and qualities. It has been working very well for us all these years, but Nix takes these, and additional concepts, much further.

Nix is a functional programming language that lets you define how to build software in a declarative fashion. By making use of a global cache, you only have to compile things that are unique to your environment, but in many cases, you can just pull the resulting software straight from this cache. This can greatly improve build times for the simple fact that there is no building *waves hand* :wave:

After a good amount of learning and experimentation, we figured out how Nix and depmake could co-exist, so that we can start making use of the new, without throwing away the old.

Future

Using Nix really opens the floodgates to potential new software and is going to make it extremely easy for us to roll out a new stack, customize it, ensure it keeps working in the same way forever, and allow multiple versions of the same software to exist alongside one another.

The bulk of the work in providing a new stack to our customers has now moved from building to testing it, which is a great improvement.

We are still considering wrapping our Nix-built software in containers and running them root-less, but this would primarily be to deliver additional security on top of the containment that we already provide.

Will we ever deprecate our old versions? Yes. But we'll do so with fairly lengthy grace-periods during which we gently nudge people towards the new.

Without further ado.. version two!

We are quite proud to present to you our new ImageMagick stack nicked v2.0.3 (not SemVer, any change can potentially break backwards compatibility). It has been in private beta for two weeks already and will now enter public beta. It is based on the latest ImageMagick version that Nix provides but with custom additions such as libraw so that our customers can keep enjoying those 131 extra formats.

Besides RAW and all the other formats that we already support, customers will be happy to learn that the new stack comes with support for both WEBP and DjVu formats.

We have updated our supported formats page so you can more easily discover differences between our stack versions. Simply click the "Compare" button for a clear overview.

For your convenience, here are the full lists:

Gained read support: 3FR, 3G2, 3GP, CANVAS, CLIP (previously write-only), FILE, FTP, HALD, HTTP, IIQ, JNX, MAC, MEF, NRW, PES, RAW, RMF, RW2, SCREENSHOT, WMF, WMZ.

Gained write support: BRF, DDS (previously read-only), INLINE (previously read-only), ISOBRL, ISOBRL6, JSON, SPARSE, UBRL, UBRL6

Gained read/write support: AAI, BGRA, BGRO, CAL, CALS, DXT1, DXT5, G4, GROUP4, HDR, JPE, JPS, MASK, MKV, PNG00, PNG48, PNG64, PSB, RGF, SIX, SIXEL, TIFF64, VIPS, WEBP

Besides more formats, our new ImageMagick version also enables HDRI (High Dynamic Range Imaging) by default, so you can expect more accurate image processing results.

Aside from these customer-facing upgrades, we're quite happy about the fact that the new version also has some operational benefits for us, such as the plugging of a number of security vulnerabilities, memory leaks, and so forth.

Notable breaking changes

Our previous version of ImageMagick had a bug in colorspace handling, resulting in darker images and the incorrectly reporting of sRGB vs RGB. This is now fixed, but if you found a way to work around this bug, you are essentially relying on it and will therefore need to uncompensate when you choose to upgrade.

We no longer support reading PGX and JPX files, and for the formats: BRG, GBR, GRB and RBG, it is now recommended to create an RGB format and then swap channels afterwards.

For reference, you can find a full changelog at the bottom of this post.

Upgrading

Transloadit recommends to start using the new ImageMagick stack for testing and non-critical workloads. You can do this by adding imagemagick_stack: "v2.0.3" in your /image/resize Robot Step.

Our apologies for taking so long to deliver on this upgrade, you can expect much timelier updates thanks to our new way of stack herding! :smile:

We'd love to hear your findings, so do let us know what you think via the well known speech bubble on our site!

Changelog

For reference, here is the full changelog:

  • Added additional checks to DCM reader to prevent data-driven faults (bug report from Hanno Böck)
  • Added final ClampToQuantum in sigmoidal colormap loop
  • Added support for languages that require complex text layout (reference)
  • Added tanh/atanh clone of legacy sigmoidal map (faster & more accurate)
  • Added define psd:additional-info to preserve the additional information in a PSD file
  • Added define psd:preserve-opacity-mask to preserve the opacity mask in a PSD file
  • Added layer RLE compression to the PSD encoder
  • Added layer RLE compression to the PSD encoder
  • Added support for GROUP4 compression to the FAX coder
  • Added support for RGB555, RGB565, ARGB4444 and ARGB1555 to the BMP encoder (reference)
  • Allowed the use of set and escapes when no images in memory (unless you attempt to access per-image meta-data). Currently does not include %[fx:...] and %[pixel:...]
  • Applied Debian patches (reference)
  • Backoff finite precision epsilon (reference)
  • Bumped Magick++ SO. Previously, a global replace changed matteColor to alphaColor
  • Can read geo-related EXIF metdata once-again (reference)
  • Check for buffer overflow in magick/draw.c/DrawStrokePolygon()
  • Coder path traversal is not authorized (bug report provided by Masaaki Chida)
  • coders/png.c: Added support for a proposed new PNG chunk (exIf read-write, eXIf read-only) that is currently being discussed on the png-mng-misc at lists.sourceforge.net mailing list
  • coders/png.c: Added support for a proposed new PNG chunk (zxIf, read-only) that is currently being discussed on the png-mng-misc at lists.sourceforge.net mailing list. Enable exIf and zxIf with CPPFLAGS="-DexIf_SUPPORTED -DxzIf_SUPPORTED" If exIf is enabled, only the uncompressed exIF chunk will be written and the hex-encoded zTXt chunk containing the raw Exif profile won't be written.
  • Correct for numerical instability (reference)
  • Correction to composite Over operator (reference)
  • Define CompositeChannels mask to Red, Green, Blue, Alpha, and Black
  • Deny indirect reads by policy, remove policy to permit, e.g., convert caption:@mytext.txt
  • Distort no longer converts grayscale image to sRGB (reference)
  • Do not close path for linejoins of round (reference)
  • Document behavior change in the security policy (thanks to yoya)
  • Don't interpret -fx option arguments (reference)
  • Don't return a zero bounding box for QueryMultilineFontMetrics() (reference)
  • Don't set background for transparent tiled images (reference)
  • Don't set update trait on alpha channel (private e-mail concerning -levels-colors option)
  • Don't sync pixel cache in AcquireAuthenticCacheView() (bug report from Hanno Böck)
  • Eliminate compiler warning
  • Enable alpha channel if background color is non-opaque (reference)
  • Evaluate lazy pixel cache morphology to prevent buffer overflow (bug report from Ibrahim M. El-Sayed)
  • Fix compile error in opencl.c (reference)
  • Fix drawing glitch for stroke widths greater than 2 (reference)
  • Fix for possible security vulnerabilities (reference)
  • Fix GetNextToken() off by one error
  • Fix memory leak in the MPC format
  • Fix MVG stroke-opacity (reference)
  • Fix pixel cache on disk regression (reference)
  • Fix possible buffer overflow when writing compressed TIFFS (vulnerability report from Cisco Talos, CVE-2016-8707)
  • Fix re-declaration of i (at the top, and inside a conditional)
  • Fix small memory leak (patch provided by Андрей Черный)
  • Fix stroke offset problem for -annotate (reference)
  • Fixed fd leak for webp coder (reference)
  • Fixed improper scaling of certain FITS images (reference)
  • Fixed incorrect padding calculation in PSD encoder
  • Fixed incorrect parsing with ordered dither (reference)
  • Fixed incorrect RLE decoding when reading a DCM image that contains multiple segments
  • Fixed incorrect RLE decoding when reading an SGI image (reference)
  • Fixed issue where the display window was used instead of the data window when reading EXR files (reference)
  • Fixed memory leak when creating nested exceptions in Magick++ (reference)
  • Fixed proper placement of text annotation for east / west gravity
  • Fixed reading DXT1 images with an alpha channel.
  • If a convenient line break is not found, force it for caption: (reference)
  • Implemented a private PNG caNv (canvas) chunk for remembering the original dimensions and offsets when an image is cropped. Previously we used the oFFs and vpAg chunks for this purpose, but this had potential conflicts with other applications that also use the oFFs chunk
  • Increase memory allocation for TIFF pixels (reference)
  • Initialize draw_info alpha member to OpaqueAlpha
  • Initialize index channel to get expected results from the stegano coder
  • Iterate channels over source image rather than destination (bug report from Hanno Böck)
  • Mask composite produces proper results for the convert utility (reference)
  • Monochrome images no longer have inverted colors (reference)
  • Note alpha channel when combining 4 or more images (reference)
  • Off by 1 error when computing the standard deviation (reference)
  • Off by one memory allocation (reference)
  • Patch so -kuwahara option can preserve colormapped edges
  • Permit EPT images with just a TIFF or EPS image, not both (reference)
  • Prevent buffer overflow (bug report from Max Thrane)
  • Prevent buffer overflow and other problems in SIXEL, PDB, MAP, TIFF and CALS coders (bug report from Donghai Zhu)
  • Prevent buffer overflow in BMP & SGI coders (bug report from pwchen&rayzhong of tencent)
  • Prevent buffer overflow when streaming an image (reference)
  • Prevent fault in MSL interpreter (reference)
  • Prevent memory use after free (reference)
  • Prevent possible buffer overflow when reading TIFF images (bug report from Shi Pu of MS509 Team)
  • Prevent possible shell command injection vulnerability through the authenticate parameter of the PDF, PCL and XPS coders (report from Erez Turjeman)
  • Prevent random pixel data for corrupt JPEG image (bug report from Hirokazu Moriguchi, Sony)
  • Prevent spurious removal of MPC cache files (reference)
  • Process channels independently for -channel -equalize (reference)
  • Properly auto-fit caption (reference)
  • Properly center text label (reference)
  • Properly initialze PES blocks (reference)
  • Quote passwords when passed to a delegate program.
  • Rather than replicate options into artifacts make a link from image to image_info and lookup a global option if no artifact is defined.
  • Recognize XML policy closing tags (reference)
  • Remove https delegate
  • Remove OpenMP calls from colormap update loops
  • Remove support for internal ephemeral coder
  • Renamed read_vpag_chunk_callback() function to png_user_chunk_callback() in coders/png.c
  • Render to clip mask rather than image for clip-path MVG graphics primitive
  • Replace show delegate title with image filename rather than label
  • Replaced CoderSeekableStreamFlag with CoderDecoderSeekableStreamFlag and CoderEncoderSeekableStreamFlag
  • Respect connected-components:area-threshold define (reference)
  • Respect gravity option (reference)
  • Restore -mattecolor option
  • Return correct offset for negative index for -fx option (reference)
  • Return unbiased standard deviation for image statistics (reference)
  • Revert patch that did not set update trait on alpha channel
  • RLE check for pixel offset less than 0 (heap overflow report from Craig Young)
  • Sanitize all delegate emedded formatting characters
  • Sanitize comments that include braces for the MIFF image format (reference)
  • Sanitize input filename for http / https delegates (improved patch)
  • Security improvements to TEXT coder broke it (reference)
  • Set alpha member of draw structure to OpaqueAlpha (reference)
  • Set colorspace to sRGB if -append has non-homogenous colorspaces (reference)
  • sigmoidal-contrast: Direct computation, without LUT
  • sigmoidal-contrast: Remove unnecessary initial ClampToQuantum
  • Support -region option (reference)
  • Support configure script --enable-indirect-reads option to enable indirect reads (@) in filenames
  • Support configure script --enable-pipes option to enable pipes (|) in filenames
  • Support pixel-cache and shred security policies
  • Support read-masks for the -modulate option
  • Support the compare -read-mask option
  • Support the phash:colorspaces and phash:normalize options
  • The -clone option no longer leaks memory
  • The -extent option now matches the results of IMv6 (reference)
  • The -stream option now increments the pixel pointer properly (reference)
  • The histogram coder now returns the correct extent
  • To comply with the SVG standard, use stroke-opacity for transparent strokes
  • Turn off alpha channel for the compare difference image (reference)
  • Unbreak build without JPEG support (reference)
  • Uninitialized data in MAT image format (reference)
  • Unit test pass again after small SUN image patch
  • Use CopyMagickString() rather than CopyMagickMemory() for strings
  • Validation unit test for MNG works again