This article provides instructions on how to improve ImageMagick’s handling of raw images by changing the open-source utilities that ImageMagick uses by default (UFraw on Linux and DCraw in Windows) to other utilities.
What is ImageMagick?
ImageMagick is open-source software for running a wide range of operations on images. It was first created in 1987 and has been maintained and developed since then. It has both a graphical user interface as well as command-line utilities. These command-line utilities are very powerful and can be used to programmatically edit and convert images, such as photographs, and even to programmatically create new images, such as procedurally-generated slates for video projects. Because it is open-source and has these command-line utilities, it has ended up being widely used in other open-source projects that need to handle images programmatically.
ImageMagick itself also delegates to other open-source programs when it needs to handle certain types of files. Although it natively handles a wide range of standard image files, such as JPEG and PNG, when it needs to handle raw camera files, it delegates the conversion of these files to other command-line utilities that are included in the ImageMagick installation. On Windows systems, ImageMagick delegates to DCraw. On Linux, it delegates to UFraw.
What are raw images?
These are files containing largely-unprocessed data directly from image sensor of a camera or scanner. There are many raw file formats, from CR2, to SR2, to K25. The data varies depending on the camera model and manufacturer. Because of the lack of standardization and the wide universe of different cameras on the market, these files are more complex to work with than standard formats like JPEG. And they need to be converted before they can really be used.
This is where both specific proprietary software provided by the camera manufacturer, as well as much-wider-ranging open-source software such as UFRaw and DCRaw come into play. The manufacturers’ software is intended for handling images from a specific camera, whereas the open-source projects try to cover as wide a range of cameras and formats as possible.
Issues with UFRaw and DCraw
The use of UFRaw as a delegate of ImageMagick has presented some challenges in recent years, largely due to the fact that the open-source UFRaw project has not been maintained in several years, and the project is not being updated for newer cameras. This means that these images might not look right when converted – for example, in our testing, we found images from one newer camera ended up converting into shades of pink, rather than full color.
DCraw has had more recent updates than UFraw, but DCraw’s development has also stalled within the past couple of years. In our testing, the images that converted into shades of pink in UFraw converted better in DCraw, but they still did not look right. In our specific examples, they came across full color but with a heavily yellowish tint.
Because of this, programmers and system administrators whose applications rely on ImageMagick for the conversion of raw camera files may wish to consider changing how ImageMagick handles these files. A couple of alternatives for UFraw and DCraw are Libraw and RawTherapee.
Libraw: Advantages and disadvantages
Libraw is a library, rather than a command-line utility, and ImageMagick is programmed to be able to use Libraw natively rather than delegating externally. However, ImageMagick must be compiled from the source code with certain parameters, rather than using the installer, in order to do this. Libraw’s methods are inherited from DCraw, and so, unfortunately, we have found in our testing that files from certain newer cameras that do not work well in DCraw also do not work well in Libraw. It is a step above using UFraw, however. DCraw does not work on Linux and Apple systems, and so an application that is running on Linux could do better by using Libraw, and as mentioned, Libraw is natively supported by ImageMagick.
RawTherapee: Advantages and disadvantages
RawTherapee is a utility for working with raw camera files. It has both a graphical user interface and a command-line utility. It appears to be a well-maintained project, and in our testing, it was able to handle quite well the images that did not work well in UFraw and DCraw.
ImageMagick does not work with RawTherapee out-of-the-box. However, ImageMagick does have an external delegates file (delegates.xml) where a programmer can set parameters to have ImageMagick delegate to RawTherapee (instead of to UFraw or DCraw). This can be accomplished without editing any source code and without doing a custom compile of ImageMagick.
Using Libraw instead of delegating
Because Libraw provides internal libraries, ImageMagick no longer needs to delegate to an external command-line utility. (In fact, ImageMagick will ignore instructions in its delegates.xml file for raw camera files once it is properly compiled with Libraw.)
As mentioned above, ImageMagick needs to be built from the source code in order for it to work with Libraw. The following instructions assume that you are starting with a clean system that does not already have ImageMagick or Libraw installed. Our example system is CentOS – you may need to make adjustments when using other Linux operating systems.
Building and compiling
ImageMagick requires the following dependencies for the decode delegate for certain image formats. If we want to convert the raw image into any commonly used image type, then the corresponding packages are required:
$ sudo apt-get install build-essential checkinstall
libx11-dev libxext-dev zlib1g-dev libpng12-dev
libjpeg-dev libfreetype6-dev libxml2-dev
After installing the required libraries, we can then download the LibRaw ubuntu package by using apt-get.
$ sudo apt-get install libraw-dev
In order to use LibRaw as a raw image decoder instead of other default options in ImageMagick, we need to install and build ImageMagick from the source. First, clone the source repository.
$ git clone https://github.com/ImageMagick/ImageMagick.git ImageMagick-7.1.0
Note that we should not directly configure and compile it with the default settings. Instead, we configure it and build it with modules:
$ cd ImageMagick-7.1.0
$ ./configure –with-modules
$ make
$ sudo make install
Testing
Now, try running ImageMagick, with debug information turned on, to convert a raw camera file:
magick -debug “all” [raw input filename, including extension] [output filename, including extension]
Search the console output for “libraw” to confirm that it successfully used libraw. You should not see it create a terminal command for UFraw or DCraw.
Finally, you should see a file created with the output filename that you provided to ImageMagick. This file should open successfully and display a rendition of the image that you provided.
Delegating to RawTherapee
RawTherapee offers a command-line utility, rawtherapee-cli. We can tell ImageMagick to delegate to this, rather than to UFraw or DCraw, by editing ImageMagick’s delegates.xml file. This involves more than simply pointing ImageMagick to RawTherapee. The delegates.xml file needs to contain the command-line command for RawTherapee, including variables that ImageMagick can populate on the fly.
You do not need to compile ImageMagick from the source code in order to use RawTherapee. The instructions below assume that you already have ImageMagick and RawTherapee installed and that ImageMagick is using UFraw or DCraw as a delegate (rather than using the Libraw libraries).
Adding RawTherapee to the path
You will need to add the directory containing rawtherapee-cli to the path in your system. The instructions for doing this vary by operating system, so you will need to follow instructions specific to your operating system for adding a directory to the path. Be sure to do this such that the variable persists for all sessions, not just the current session.
Finding ImageMagick’s delegates.xml
ImageMagick calls UFraw and DCraw via the terminal. It also does this with a number of other applications to which it delegates various file types, such as PDFs and Office documents. The delegates.xml file serves as a reference and configuration file for ImageMagick to build the command-line commands to call these applications with the correct parameters to handle the file that is at hand. To have ImageMagick use RawTherapee to convert raw camera files, we will need to edit the delegate decode="dng:decode" line of the delegates.xml file.
The exact location of your delegates.xml file depends on the details of your installation and your operating system. ImageMagick looks in a number of different places for this file, and it is possible to have more than one copy. You may be able to simply search your system for “delegates.xml”. However, in case there are multiple copies, it can be useful to know which one has precedence (which one ImageMagick is checking first).
To do this, we will want to run ImageMagick’s convert utility on an example file, and have it write debugging information to the console. We can just use an input file that does not actually exist and has a nonsensical extension, such as “abc.xyz”. It will error out, but we will still get the debug information. We will tell it to output to JPEG, in file test.jpg (however, it will error out before it gets there because the input file does not exist).
Run the following command:
magick -debug “all” abc.xyz test.jpg
It will log copious debugging information to the console. Search the debugging information for a line that starts with “searching for configure file”. You may see several of these lines followed by a line that starts with “loading delegate configuration file”. The path contained in this latter line tells you where your delegates.xml file is. See examples below:
Searching for configure file: C:Program FilesImageMagick-7.1.0-Q16-HDRIdelegates.xml
Loading delegate configuration file C:Program FilesImageMagick-7.1.0-Q16-HDRIdelegates.xml
Editing delegates.xml
ImageMagick’s delegates.xml file contains a number of substitution flags that it uses to populate the command with the actual parameters. The file contains a list of these at the top. For example, %i is the input filename, and %u is the unique temporary filename. For our purposes, we’ll be using a substitution flag that is not actually referenced in that list. This one is %F, which is the original filename with the suffix.
When calling a raw image conversion delegate, ImageMagick first creates a temporary copy of the original file without a suffix. It then provides the delegate with a unique, temporary output filename. ImageMagick takes that output file and then processes it further as needed to create whatever rendition is needed.
The challenge is that RawTherapee requires the input file to have a suffix, and so we cannot use the temporary copy of the original file (unless we were to edit the ImageMagick source code to include a suffix in this filename). If we do not want to edit the ImageMagick source code, then we can simply structure the delegate line so that this temporary input file is ignored and instead we simply use the original file as the input. That is why we use the %F substitution flag.
We structure the delegate line to call the application rawtherapee-cli, which should be on your system’s path. We use the -o parameter in RawTherapee to specify the output filename (which should be %u.png). And we use -c to specify the input file (which should be %F). The parameter -n tells RawTherapee to output as a compressed PNG, and -Y overwrites the output file if that file is already present (although, since ImageMagick generates a unique filename for the requested output file, this should not actually ever occur).
We use " to specify quotes within the terminal command.
Thus, the delegate line is as follows:
<delegate decode=”dng:decode” command=”"rawtherapee-cli" -o "%u.png" -n -Y -c "%F"”/>
Testing
Now, try running ImageMagick, with debug information turned on, to convert a raw camera file:
magick -debug “all” [raw input filename, including extension] [output filename, including extension]
If you analyze the console output, you should see a command such as follows:
rawtherapee-cli -o [unique temporary output filename with a .png extension] -n -Y -c [the raw input filename that you provided to ImageMagick]
You should also see RawTherapee run that process and print information to the console.
Finally, you should see a file created with the output filename that you provided to ImageMagick. This file should open successfully and display a rendition of the image that you provided. If the input file was from a newer camera, the output rendition may be much more correct and faithful to the original than that provided by UFraw or even DCraw or Libraw.
Want more information?
Are you interested in learning more? iSoftStone has deep experience around Content Services Platforms. Please reach out to talk more about our experience with ImageMagick and how to integrate it into a Nuxeo pipeline.
