I’ve had a long-standing to-do item in the MagicScaler codebase, which was to add a configuration option to force embedding an sRGB ICC profile in output images or to tag output images with the sRGB colorspace Exif tag. I had assumed that at some point, someone would ask for such a thing or would report an issue that turned out to be related to improper colorspace interpretation in another bit of software, which could be fixed by embedding or tagging the profile. Surprisingly, nobody ever did.

MagicScaler has always converted images to sRGB on input and saved its output as sRGB, because sRGB is the colorspace of the Web, and MagicScaler’s primary intended use is Web output. Web browsers and other common software have a spotty history when it comes to color management support, and most of the ones that don’t do color management simply assume that everything is sRGB. Or they don’t even know what sRGB is and just let the OS or hardware handle colors, meaning they likely get sRGB anyway. Furthermore, most W3C specs related to colors either require sRGB explicitly or specify that in the absence of evidence to the contrary, all colors should be treated as sRGB. The general idea is, make everything sRGB, and you never have to worry about colorspaces again (on the web at least – until we all have HDR monitors and are enjoying our 12-bit JPEGs). For the most part, it’s true… which I assume is why nobody ever asked for anything different.

A few weeks ago, however, I received a request to add an option to MagicScaler to allow it to skip its internal sRGB working-space conversion and keep the image in its original colorspace, embedding the source ICC profile in the output image. In general, that’s a bad idea, because most of MagicScaler’s algorithms assume they’re working with sRGB (or sRGB-like) data. But the person who made the request had an interesting use case, so I decided to combine that effort with my other to-do item.

Why embed sRGB?

If the Web is all sRGB all the time, why bother with the profile? Shouldn’t an image without a profile be the same as one with the sRGB profile as far as any web software is concerned? Maybe not…

There were two main reasons I had put that item on my list in the first place. One was a scary warning I often saw when using Jeffrey Friedl’s online Image Metadata Viewer

WARNING: No color-space metadata and no embedded color profile: Windows and Mac web browsers treat colors randomly.

Images for the web are most widely viewable when in the sRGB color space and with an embedded color profile. See my Introduction to Digital-Image Color Spaces for more information.

The other was that I remembered reading a post by Ryan Mack from the Facebook Engineering team a few years ago abut their TinyRGB (c2) sRGB-compatible ICC profile.

Going back to 2012, Facebook has been embedding its TinyRGB profile in every thumbnail and resized JPEG it serves. This extra 524-byte profile has been tacked on to billions of images and likely served hundreds of billions of times. In the post, he explains that they noticed on certain computers/devices that had a display colorspace other than sRGB, some web browsers would treat images as if they were encoded in the display colorspace rather than sRGB. If the display had a wide-gamut colorspace configured, colors in images would be oversaturated/overblown.

I have personally never experienced those types of issues, but I’ve also never used a fancy profiled wide-gamut monitor, so I guess I wouldn’t have.

Anyway, web browsers have come a very long way since 2012 in terms of color management support, and I wondered whether this is still an issue at all. But I just grabbed a thumbnail of a photo recently posted to Facebook, and they’re still embedding that same TinyRGB profile 6 years later. I’d assume Facebook would be pretty happy to cut 524 bytes off every JPEG they serve if they could do so with no ill effects.

Looking into it further, I found a great description of the problem broken down by OS and browser. The linked post indicates that this is a still problem as of its last update in July 2017.

So apparently, it’s still an issue, and I reckon I ought to do something about it. The solution recommended in each case is to assign the sRGB profile to images that don’t have a profile attached. But the standard sRGB profile attached to most images (and the one included in Windows) is just over 3KB, and that’s a lot of overhead to correct an issue that affects only a small percentage of users,

It was pretty cool, then, that the Facebook engineers were able to create a compatible profile so much smaller. I figured I’d probably want to use their tiny profile as well to keep the overhead down. However, as I was looking into the copyright/license status of their profile to see if I’d be allowed to embed it in MagicScaler, I ran across an interesting post by Øyvind Kolås (hereafter referred to by his twitter handle, @Pippin), who claimed to have created an even tinier (487-byte) sRGB-compatible profile, which he called sRGBz.

Thus began my own investigation into ICC profile optimization and my own effort to make a better, smaller sRGB-compatible profile. This led me down a deep rabbit hole, where I learned a ton, and I thought I’d document what I learned here. There was so much, I’ll have to split it into multiple posts.

Trim the Fat

If you’re not familiar with how profiles work or all the many, many things that can be wrong with them, I highly recommend Elle Stone’s articles on color management for some background. Color management is a tricky subject, and I’ve learned a ton from her site.

I’ll also be referring quite a bit to the specification for v2 ICC profiles, because ultimately, I want to abuse the spec to save those precious, precious bytes… but I want to do so in a completely compatible way.

An ICC profile consists of three main parts

1. A 128-byte header. This is fixed in size, and although it contains some empty reserved padding, there’s nothing that can be done to save space here that won’t break many/most profile readers.
2. A directory of tags (records) in the profile. Each directory entry consists of a 4-byte tag identifier, a 4-byte offset to the start of the tag data, and a 4-byte length for the tag data. That’s 12 bytes per tag for those keeping track, so the fewer tags the better (duh).
3. The tag data. Each tag starts with an 8-byte tag header, which consists of a 4-byte identifier and 4-bytes of reserved space. The actual tag content follows. Some tags are fixed-length, some are variable. And each tag must start on a 4-byte boundary, so there may be alignment issues that cause wasted space.

Any effort to save space will be constrained by that structure and by the tags required for each profile type. According the spec, RGB profiles require a minimum of 9 tags: description (desc), copyright (cprt), white point (wtpt), red, green and blue primary values (rXYZ, gXYZ, bXYZ), and red, green and blue tone reproduction curves (rTRC, gTRC, bTRC).

As Pippin correctly points out in his post, the black point (bkpt) tag included in the TinyRGB profile is not explicitly required. In fact, the ICC now explicitly recommends against using it. Plus, its data is completely redundant. In a well-behaved profile black will be defined as X=0, Y=0, Z=0, as it is in the standard sRGB profile. In the absence of a black point tag, the ICC v2 spec clearly says it is to be assumed to be (0,0,0). So we can very safely omit that tag. That saves 12 bytes for the tag directory entry, 8 bytes for the tag header and 4 bytes each for the X, Y, and Z values, for a total of 32 bytes. Minus that tag, Facebook’s TinyRGB profile could easily have been 492 bytes instead of 524.

The other space-saving change Pippin made was to reduce the length of the profileDescriptionTag and move it to the end to eliminate its effect on tag alignment. He claimed that by reducing the description to a single character (z) from Facebook’s 2-character name (c2), he could save the one byte, plus another 4 from the alignment, making a 5-byte reduction. That didn’t add up for me, given that ICC profiles use 4-byte alignment, there’s no way for alignment to waste more than 3 bytes. Since that sounded fishy, I loaded up both the 487-byte and 491-byte versions of sRGBz in the ICC Profile Dump Utility and validated them. They both reported the following:

NonCompliant! - profileDescriptionTag - ScriptCode must contain 67 bytes.

That sent me back to the spec to dig in to the structure of the profileDescriptionTag. It is defined as a complex structure that contains the description in 3 different formats: 7-bit ASCII, Unicode, and ScriptCode. The ASCII description is to be treated as the canonical name of the profile and is required; the other two are optional. In case, like me, you’ve never heard of ScriptCode, it appears to be a thing from Mac OS (the old obsolete one, not OS X).

The length/structure of the tag is as follows:

1. 8-byte header
2. 4-byte length of the ASCII description (including null terminator)
3. ASCII data of variable length -- at least one printable character, plus the null
4. 4-byte Unicode language code
5. 4-byte Unicode description length
6. Unicode description of variable length -- can have length of 0
7. 2-byte ScriptCode code
8. 1-byte ScriptCode description length
9. 67 bytes reserved for ScriptCode data

I couldn’t even begin to guess the reason behind a fixed-length reserved space for the ScriptCode data when the others are variable-length, but that’s what the validator was complaining about. If we assume both the Unicode and ScriptCode descriptions will be empty, the length of the description tag will be 8 + 4 + 4 + 4 + 2 + 1 + 67 = 90 bytes, plus the length of the ASCII string, plus its null terminator. That would be 92 bytes for 1-character description or 96 for a 5-character description. Those are incorrectly listed as 91 and 95 bytes in the sRGBz-487 and sRGBz profiles, respectively, and the files are 1-byte short each. So they are, in fact, not valid.

Interestingly, if you add an extra byte to the profile without adjusting the length of the description tag, the validator doesn’t complain. It’s only because the tag is at the end of the file and there’s no padding before another aligned tag that the validator has an issue.

That prompted me to look at the TinyRGB/c2 profile to see where the math went wrong, and it turns out theirs is wrong too. They have the description length listed as 94 bytes, but it really should only be 93. They include the description tag early and pad it out to 96 bytes for alignment, which is enough to satisfy the ICC validator tool, but it looks like it might have caused issues in certain versions of Adobe Illustrator.

In any case, they could have fit 3 more characters in the description for no extra space cost had they wished.

Anyway, after correcting the description tag lengths in the sRGBz profiles, they come out to 488 bytes for the minimal 1-character-name version and 492 for the friendly-named version, same as TinyRGB minus the black point tag.

But we can do better. Quite a bit better, actually…

Abuse the Spec

Pippin mentions in his post that he experimented with packing some tag data in the 44 bytes of reserved padding of the profile header but that it didn’t work out. So, while that’s not an option, there’s another even larger bit of padding that we can put some data into: the 67 bytes reserved for the ScriptCode description. As a test, I chose to move the tone reproduction curve data, which just happens to be 64 bytes. It’s perfectly legal for tag data to overlap, and in fact, for the TRC tags, it’s expected. Well-behaved RGB profiles should have identical curves in the red, green, and blue TRC tags, and it’s common for the three directory entries to refer to a single copy of the data for all of these. This is the case in the standard HP/Microsoft sRGB profile (which would be 4K larger otherwise) and in the TinyRGB profile. If we move that tag data to overlap the ScriptCode reserved area, we can save the full 64 bytes.

As for whether that’s safe, I’ll say the following:

1. ScriptCode is a Mac OS thing, which is to say it’s not a thing anymore. Nobody will ever be looking at that area for ScriptCode.
2. The profileDescriptionTag has a 1-byte ScriptCode length field to indicate how many of the 67 reserved bytes contain description data. We set that to 0, so even if some software did read that section of the tag, it shouldn’t go on to read any of the data.
3. Although the spec does explicitly say that unused bytes in the ScriptCode area should be set to 0, no software I’ve encountered has had any problem with that area containing the TRC data, and all software should be fine with the TRC tag data not having its own dedicated space.

That means we can cut the TinyRGB profile down to 428 bytes simply by removing the black point tag and relocating the TRC data. Finally, if we’re clever with the alignment, we can shave another 4 bytes off. Remember I said that the TinyRGB profile had its description tag length wrong? Well, if we correct that, we can save 1 byte, and it had 2 bytes of padding to align the tag that follows (the copyright tag in their case). Plus, we still have 3 unused bytes left over from the 67-byte ScriptCode area.

The ScriptCode area is tricky because the position of that section is dependent on the length of the ASCII description. Since we have to align the start of the description tag on a 4-byte boundary, if we were to use a minimum 1-character ASCII description, the ScriptCode data section would start at an offset of 25 from there, leaving the first 3 bytes unusable because we can’t start a new tag until offset 28. That means wasting the first 3 of those 67 as padding. That would still allow us to use the last 64 bytes to hold the TRC tag data, though, and the alignment would be correct to start the next tag immediately after.

OR… we could use three extra description characters to give a more descriptive name and have the 67 bytes start on a 4-byte boundary. I chose that option, making the description ‘c2ci’ to differentiate it from the original. That allows the 64 bytes of the TRC tag to start at the beginning of the ScriptCode block and leaves the last 3 for the start of the next tag.

Overall, the length of the description tag ends up being 95 bytes, but as far as the alignment of the following tags go, it doesn’t matter, because they overlap. It’s as if the length is actually 28, which was the offset at which we started the curve data. That 28, plus the 64 of the TRC allows the next tag to start at offset 92, meaning we saved 4 bytes over the 96-byte alignment that Facebook used.

There’s one last place that space could be saved if we were so inclined. Facebook used ‘FB’ for their copyright text but then had to include a byte of padding because that results in an 11-byte tag. If we moved the copyright tag to the end of the file, we wouldn’t need that padding, because there’s no need to align for another tag. That would make the final size 423 bytes. I liked the change Pippin made in his sRGBz profile, though, which was to set the copyright text to ‘CC0’ – a value that fits perfectly in a 12-byte tag. Facebook has since released their profile under the CC0 license, so that’s a good change to make in my alternate.

And that’s my compact profile starting point. At 424 bytes (an even 100-byte savings from the original) it can have the exact same data as TinyRGB/c2 -- minus the redundant black point tag, plus some extra description characters and corrected copyright text. Here’s that file for reference if you want to check it out. But let me say, you won’t want to use it for anything real. I’m going to do much better before I’m done.

Not Just Tinier – Better

So what’s wrong with the TinyRGB or its new tinier variant? A couple of things, actually…

I’ve mentioned well-behaved RGB profiles a couple of times now, and if you didn’t follow the link to Elle Stone’s post on the subject, I highly recommend you do that. Pippin mentions in his sRGBz post that he improved the matrix precision of his profile, and what that means is that his profile was created using XYZ color values that are balanced to allow for properly-neutral grey colors. TinyRGB uses the unbalanced values from the old HP/Microsoft sRGB profile. I’ll be ensuring I’ve got the most correct values possible in my profile.

And, like Pippin, I was curious about that 26-point TRC tag Facebook came up with. It turns out, that’s not all that great either.

I’ll have entire posts on both of those topics, because I made some fascinating (to me at least) findings in researching and testing them. Tune in next time for my post on finding the perfect curve…

Well, they’ve gone and done it. The corefx team has finally acquiesced to the many requests that they include System.Drawing in .NET Core.

The upcoming System.Drawing.Common package will include most of the System.Drawing functionality from the full .NET Framework and is meant to be used as a compatibility option for those who wish to migrate to .NET core but were blocked by those dependencies. From that standpoint, Microsoft is doing the right thing. Reducing friction as far as .NET Core adoption is concerned is a worthy goal.

On the other hand, System.Drawing is one of the most poorly implemented and most developer-abused areas of the .NET Framework, and many of us were hoping that the uptake of .NET Core would mean a slow death for System.Drawing. And with that death would come the opportunity to build something better.

For example, the mono team have released a .NET-compatible wrapper for the Skia cross-platform graphics library from google, called SkiaSharp. Nuget has come a long way in supporting platform-native libraries, so installation is simple. Skia is quite full-featured, and its performance blows System.Drawing away.

The ImageSharp team have also done tremendous work, replicating a good deal of the System.Drawing functionality but with a nicer API and a 100% C# implementation. This one isn’t quite ready for production use yet, but it appears to be getting close. One word of warning with this library, though, since we’re talking about server apps: As of now, its default configuration uses Parallel.For internally to speed up some of its operations, which means it will tie up more worker threads from your ASP.NET thread pool, ultimately reducing overall application throughput. Hopefully this will be addressed before release, but it only takes one line of code to change that configuration to make it server-friendly.

Anyway, if you’re drawing, graphing, or rendering text to images in a server-side app, either of these would be worth a serious look as an upgrade from System.Drawing, whether you’re moving to .NET Core or not.

For my part, I’ve built a high-performance image processing pipeline for .NET and .NET Core that delivers image quality that System.Drawing can’t match and that does it in a highly scalable architecture designed specifically for server use. It’s Windows only for now, but cross-platform is on the roadmap. If you use System.Drawing (or anything else) to resize images on the server, you’d do well to evaluate MagicScaler as a replacement.

But the resurrection of System.Drawing, while easing the transition for some developers, will probably kill much of the momentum these projects have gained as developers were forced to search for alternatives. Unfortunately, in the .NET ecosystem, a Microsoft library/package will almost always win out over other options, no matter how superior those alternatives might be.

This post is an attempt to make clear some of the shortcomings of System.Drawing in the hopes that developers will evaluate the alternatives even though System.Drawing remains an option.

I’ll start with the oft-quoted disclaimer from the System.Drawing documentation. This disclaimer came up a couple of times in the GitHub discussion debating System.Drawing.Common.

"Classes within the System.Drawing namespace are not supported for use within a Windows or ASP.NET service. Attempting to use these classes from within one of these application types may produce unexpected problems, such as diminished service performance and run-time exceptions"

Like many of you, I read that disclaimer a long time ago, and then I went ahead and used System.Drawing in my ASP.NET apps anyway. Why? Because I like to live dangerously. Either that, or there just weren’t any other viable options. And you know what? Nothing bad happened. I probably shouldn’t have said that, but I’ll bet plenty of you have had the same experience. So why not keep using System.Drawing or the libraries built around it?

Reason #1: GDI Handles

If you ever did have a problem using System.Drawing on the server, this was probably it. And if you haven’t yet, this is the one you’re most likely to see.

System.Drawing is, for the most part, a thin wrapper around the Windows GDI+ API. Most System.Drawing objects are backed by a GDI handle, and there are a limited number of these available per process and per user session. Once that limit is reached, you’ll encounter out of memory exceptions and/or GDI+ ‘generic’ errors.

The problem is, .NET’s garbage collection and finalization process may delay the release of these handles for long enough that you can overrun the limit even under relatively light loads. If you forget (or don’t know) to call Dispose() on objects that hold one of those handles, you run a very real risk of encountering these errors in your environment. And like most resource-limit/leak bugs, it will probably get missed during testing and only bite you once you’ve gone live. Naturally, it will also occur when your app is under its heaviest load, so the max number of users will know your shame.

The per-process and per-session limits vary by OS version, and the per-process limit is configurable. But no matter the version, GDI handles are represented with a USHORT internally, so there’s a hard limit of 65,536 handles per user session, and even well-behaved apps are at risk of encountering this limit under sufficient load. When you consider the fact that more powerful servers allow us to serve more and more concurrent users from a single instance, this risk becomes more real. And really, who wants to build software with a known hard limit to its scalability?

Reason #2: Concurrency

GDI+ has always had issues with concurrency, and although many of those were addressed with architectural changes in Windows 7/Windows Server 2008 R2, you will still encounter some of them in newer versions. Most prominent is a process-wide lock held by GDI+ during any DrawImage() operation. If you’re resizing images on the server using System.Drawing (or the libraries that wrap it), DrawImage() is likely at the core of that code.

What’s more, when you issue multiple concurrent DrawImage() calls, all of them will block until all of them complete. Even if the response time isn’t an issue for you (why not? do you hate your users?), consider that any memory resources tied up in those requests and any GDI handles held by objects related to those requests are tied up for the duration. It actually doesn’t take very much load on the server for this to cause problems.

There are, of course, workarounds for this specific issue. Some developers spawn an external process for each DrawImage() operation, for example. But really, these workarounds just add extra fragility to something you really shouldn’t be doing in the first place.

Reason #3: Memory

Consider an ASP.NET handler that generates a chart. It might go something like this:

1. Create a Bitmap as a canvas
2. Draw some shapes on that Bitmap using Pens and/or Brushes
3. Draw some text using one or more Fonts
4. Save the Bitmap as PNG to a MemoryStream

Let’s say the chart is 600x400 pixels. That’s a total of 240,000 pixels, multiplied by 4 bytes per pixel for the default RGBA format, so 960,000 bytes for the Bitmap, plus some memory for the drawing objects and the save buffer. We’ll call it 1MB for that request. You’re probably not going to run into memory issues in this scenario, and if you do, you might be bumping up against that handle limit I mentioned earlier because of all those Bitmaps and Pens and Brushes and Fonts.

The real problem comes when you use System.Drawing for imaging tasks. System.Drawing is primarily a graphics library, and graphics libraries tend to be built around the idea that everything is a bitmap in memory. That’s fine if you’re thinking small. But images can be really big, and they’re getting bigger every day as high-megapixel cameras get cheaper.

If you take System.Drawing’s naive approach to imaging, you’ll end up with something like this for an image resizing handler:

1. Create a Bitmap as a canvas for the destination image.
2. Load the source image into another Bitmap.
3. DrawImage() the source onto the destination, resized/resampled.
4. Save the destination Bitmap as JPEG to a MemoryStream.

We’ll assume the same 600x400 output as before, so we have 1MB again for the destination image and Stream. But let’s imagine someone has uploaded a 24-megapixel image from their fancy new DSLR, so we’ll need 6000x4000 pixels times 3 bytes per pixel (72MB) for the decoded RGB source Bitmap. And we’d use System.Drawing’s HighQualityBicubic resampling because that’s the only one that looks good, so we need to add another 6000x4000 times 4 bytes per pixel for the PRGBA conversion that it uses internally, making another 96MB. That’s 169MB(!) for a single image resizing request.

Now imagine you have more than one user doing the same thing. Now remember that those requests will block until they’re all complete. How many does it take before you run out of memory? And even if you’re not concerned about running completely out of memory, remember there are lots of ways your server memory could be put to better use than holding on to a bunch of pixels. Consider the impact of memory pressure on other parts of the app/system:

• The ASP.NET cache may start dumping items that are expensive to re-create
• The garbage collector will run more frequently, slowing the app down
• The IIS kernel cache or Windows file system caches may have to remove useful items
• The App Pool may overrun its configured memory limit and get recycled
• Windows may have to start paging memory to disk, slowing the entire system

None of those are things you want, right?

A library designed specifically for imaging tasks will approach this problem in a very different way. It has no need to load either the source or destination image completely into memory. If you’re not going to draw on it, you don’t need a canvas/bitmap. It goes more like this:

1. Create a Stream for the output JPEG encoder
2. Load a single line from the source image and shrink it horizontally.
3. Repeat for as many lines from the source as required to create a single line of output
4. Shrink intermediate lines vertically and write a single output line to the encoder
5. Goto 2. Repeat until all lines are processed.

Using this method, the same image resizing task can be performed using around 1MB of memory total, and even larger images incur only a small incremental overhead.

I know of only one .NET library that is optimized in this way, and I’ll give you a hint: it’s not System.Drawing.

Reason #4: CPU

Another side-effect of the fact that System.Drawing is more graphics-focused than imaging-focused is that DrawImage() is quite inefficient CPU-wise. I have covered this in quite a bit of detail in a previous post, but that discussion can be summarized with the following facts:

• System.Drawing’s HighQualityBicubic scaler works only in PRGBA pixel format. In almost all cases, this means an extra copy of the image. Not only does this use (considerably) more RAM, it also burns CPU cycles on the conversion and the processing of the extra alpha channel.
• Even after the image is in its native format, the HighQualityBicubic scaler performs roughly 4x as many calculations as are necessary to obtain the correct resampling results.

These facts add up to considerable wasted CPU cycles. In a pay-per-minute cloud environment, this directly contributes to higher hosting costs. And of course your response times will suffer.

And think of all the extra electricity wasted and heat generated. Your use of System.Drawing for imaging tasks is directly contributing to global warming. You monster.

Reason #5: Imaging is deceptively complicated

Performance aside, System.Drawing doesn’t get imaging right in many ways. Using System.Drawing means either living with incorrect output or learning all about ICC Profiles, Color Quantizers, Exif Orientation correction, and many more domain-specific topics. It’s a rabbit hole most developers have neither the time nor inclination to explore.

Libraries like ImageResizer and ImageProcessor have gained many fans by taking care of some of these details, but beware, they’re System.Drawing on the inside, and they come with all the baggage I've detailed in this post.

Bonus Reason: You can do better

If, like me, you’ve had to wear glasses at some point in your life, you probably remember what it was like the first time you put them on. I thought I could see ok, and if I squinted just right, things were pretty clear. But then I slid those glasses on, and the world became a lot more detailed than I knew it could.

System.Drawing is a lot like that. It does ok if you get the settings just right, but you might be surprised how much better your images could look if you used a better tool.

I’ll just leave this here as an example. This is the very best System.Drawing can do versus MagicScaler’s default settings. Maybe your app would benefit from getting glasses…

System.Drawing

MagicScaler

Photo by Jakob Owens

So look around, evaluate the alternatives, and please, for the love of kittens, stop using System.Drawing in ASP.NET.