The color space of demosaiced raw data

In my new article "The Canon sRaw and mRaw Output Formats", when I line out the conceptual stages of "development" of the sensor raw data into a JPEG image file, I mention "transformation of primaries". I don't elaborate, as I was on my way to quite enough complication. But I thought I would talk a little about that now.

After a "pure" demosaicing process (whether or not such really happens), we have an image represented in terms of an r, g, and b value for each pixel. (I use lower-case letters so as not to imply that these are the R, G, and B of any familiar color space, and further to remind us that these values are nominally linear - have not had "gamma-precompensation" applied).

But they clearly seem to be the coordinates of some form of "RGB-family" color space (where here the use of the upper case is just the usual way of spelling the family name). Any such "tristimulus" color space implies a particular set of primary chromaticities ("primaries"). If we make up some light by combining "amounts" of these primaries as dictated by the coordinate values, we will get light having the specified color. By that I mean that it will look to the human eye to be that color, and that is in fact its color, by definition. ("if it honks like a horn, it is a horn.")

So, then, what are the primaries of the "color space" of the demosaiced "rgb" representation?

Well, we can't answer that. The reason is that this representation is not actually a color space. By that I mean that the set of r, g, and b values for a pixel (lets assume "perfect" demosaicing) will not assuredly represent the color for the light at that part of the sensor image, under any color space definition we might contrive.

Here's why. For a "three channel" pixel detector (which we are imagining we have, by assuming perfect demosaicing) to actually tell us, in every case, the color of the light on it requires that the three "subdetectors" have spectral responses that follow the implied spectral responses of the three "channels" of the human eye. (These are the so-called x, y, and z functions of wavelength specified by the CIE.) Only then will the three outputs describe the color, regardless of the spectral distribution of the light.

If a certain incandescent source and a certain fluorescent source look to the eye to have the same color (and if so, they are the same color), then only if that condition is met can we be assured that the three output values of our pixel detector will be the same for both sources. [I'll defer for now the question of "what color space is that, anyway", since as we'll see in just a second, it isn't really pertinent.]

In actual sensor design, the z, y, and z spectral response curves are not normally used. There are a number of reasons, but the one that is most easily understood is that doing so would give an unattractively-low overall sensitivity to the sensor. (We can of course overcome that with more gain, but then the noise goes up, etc. etc. etc.)

As a result, the the outputs do not consistently exactly describe the color of the light under any color space definition we might contrive to accommodate the realities.

Now suppose that we had a laboratory display whose three primaries could be set to any chromaticity we wanted. Could we choose primaries such that, if the three primaries were "excited" by the r, g, and b values out of our "pixel detector", the generated light would always have the color of the light we started with on the sensor. No.

Thus, the "rgb" outputs of our ideal pixel detector (or, in our usual situation, of the demosaiced sensel outputs) do not truly operate in any color space, and there is no associated set of primaries.

Now realizing that we are dealing with a compromised sensory organ, can we find a set of primaries to put into effect in our laboratory display such that in most cases, for light having a "not too bizarre" spectral distribution, the color generated by the display is very close in color to the color of the light? Probably.

Then we can think of those primaries as being the "effective native primaries" of the rgb representation of our demosaiced data (subject to the various weasel words just stated).

Now, if we have told our camera to deliver its developed image in the sRGB color space, is it likely that these "effective native primaries" are the same as the primaries upon which the sRGB color space is predicated? Not likely.

So we must, someplace along the development chain, transform the representation from one based on the effective native primaries to one based on, say, the sRGB primaries. And this in fact is a very straightforward mathematical operation.

It is is probably consolidated with other parts of the development chain. But it is important to recognize that it exists.

What in fact are these "effective primary chromaticities" for, say, a certain Canon dSLR? I don't know. But there is a tantalizing "hint" in my work on the Canon sRaw format that they might be (surely by intent) essentially the primaries of the "NTSC RGB" color space, essentially the primaries assumed by the US analog TV transmission scheme.

********

Now, lets return to our "ideal" sensor, in which the three "channels" in fact have spectral sensitivities in accordance with the x, y, and z curves.

Would its outputs unfailingly describe the color of the light under some color space. Yes.

What color space would that be? It would be the CIE XYZ color space, where the three "channel" outputs would be the values X, Y, and Z. (That is in fact the origin of the definition of this color space.)

Given that this is a "tristimulus" type of color space (in fact, often the term "tristimulus color space" is used to mean this particular one, although it applies equally well to all the RGB color spaces), what primaries is it predicated on?

The answer: primaries identified as X, Y, and Z.

Great. What are their chromaticities?

Well, here it gets a bit sticky, These primaries (kinds of light) are in fact not kinds of visible light, nor can they in any way physically be created as any sort of electromagnetic radiation (not even in the infrared or ultraviolet regions, for example).

They are mathematical fictions. On the familiar CIE x-y chromaticity chart, they lie at the origin and along the two axes, all well outside the locus of visible chromaticities.

But if we did have an "ideal" sensor, whose three "channel" outputs were in fact values of the coordinates X, Y, and Z, we could easily, by a straightforward mathematical operation, change that description into one predicated on "real" primaries, such as those of the sRGB color space.

In fact that transformation, and the inverse one, takes place all the time "under our feet". The XYZ color space (based on these three "imaginary" primaries) is in fact the lingua franca* used in color space transformation via profiles.

That is, if, in an image processor, we take in a file using the Adobe RGB color space, but want to work with the file with colors defined under the sRGB color space, then (in effect), each pixel first has its color representation changed from the Adobe RGB color space to the CIE XYZ color space, and then changed from that to the sRGB color space, in each case involving (among other things) a transformation of primaries. (In reality, the two mathematical operations are often combined and applied in one step. But the concept is as I have described.)

********

Well, aren't you glad you asked.

In my original "report" on this matter, I said:


It would have been more accurate to have said:


That requirement is said to be conformity with the "Luther-Ives" conditions.

In the latter aspect of this (the "linear combinations") clause, when we have such a sensor, the X, Y, and Z coordinates of the color indicated can be explicitly retrieved by the application of a matrix which, in effect, backs out the effect of the two "foreign" components of each response curve (that is, the y and z curves that form part of the response curve of the "X" channel).

Best regards,

Doug