Doug Kerr
Well-known member
This is to report the conclusion of a long trail of investigation, which has in part been chronicled in another thread.
**************
Common wisdom has been that in the Canon raw file (.CR2), the sensel values are recorded with a (fixed) offset (1024 in the 14-bit bodies). Thus, the binary value 1024 actually represented a sensel value of 0. The offset here can be regarded as a creature of teh numbering system.
The purpose of doing so is ostensibly to allow for negative sensel values. While such a value would have no meaning photometrically, it can certainly occur in reality as a result of several factors, including variations in the "black level" of the individual sensel and read noise.
This offset value can be thought of as a "black level" for the raw file - a value of that number being considered (for better or worse) to represent "black". (This is not the same thing as the "black point" we may adopt when doing tonal scale optimization.) This is then not so much a creature of the numbering system as a property of the "system".
It now seems as if it is not that simple, at least in the CR2 files of the EOS 40D. Rather, it seems that the Canon CR2 raw file contains a separate black level value for each row of sensels. Ordinarily, these are all in the general region of 1024.
My suspicion is that this is a way to reflect variations in the black levels of individual sensels without determining same for each sensel (and recording that in the file). It may well be that under the way the sensor is read out, the principal causes of black level discrepancy and even read noise may be row-oriented.
Now, I have not actually been able to find this "table" in the CR2 file. I suspect it is held in the MakerNote area in a proprietary format not yet known to me.
But a "tag" (data area) for just thing occurs in the DNG ("digital negative") file - a format developed by Adobe to provide an "industry standard" raw file. If I have Photoshop CS5 convert a Canon CR2 file to a DNG file, and examine the metadata in that file, I find exactly what I describe above. (The tag is called BlackLevelDeltaV - it is part of a broader scheme of recording black levels.)
There is in fact exactly that tag (and its associates) in the current standard for Exif Metadata. I suspect this is a "back formation" from the DNG specification, but I don't know. In any case, the Canon CR2 files do not seem to carry that tag (or the other ones needed in connection with it). Almost certainly, Canon developed their format before these tags were added as bona fide Exif tags.
There are still some mysteries in this area. The raw file analysis program Rawnalyze states, for a Canon CR2 file, the range of black level values in the file in terms of minimum, maximum, and average. For the files I have examined, these values do not agree well at all with the values I fond in the BlackLevelDeltaV tag in the derived DNG file. There could be various explanations for this. Sadly, Gabor Schorr, the developer of Rawnalyze, is no longer with us, so I cannot ask for his help in understanding this.
I am doing further work in this regard.
Of course, there are those who know just how this works, and I have inquired of several of them. I have not yet received any replies. Some colleagues whom I thought might know report that they don't, but would be glad to hear from when when I have found out.
In one typical CR2 file I examined, the range of the 2596 black level values is from 1020.542969 to 1029.398437 (they are carried in 64 bits as a "signed rational"), with an average of 1026.478509.
Some more detail
Here is a little more information on the provision for black levels in the DNG file specification. (It is paralleled in the current Exif metadata specification.)
a. It is possible to state a black level for each sensel.
b. As an alternative it is possible to state a "base" black level pertaining to all sensels regardless of "channel" (R, G, B).
c. As an alternative it is possible to state a "base" black level each channel, applicable to all sensels of the channel.
d. It is then possible to define a "differential" black level by row; this is added to the "base" black level mentioned above for all sensels in the row.
e. It is also possible to define a "differential" black level by column; this is added to the "base" black level mentioned above for all sensels in the column.
f. Potentially, both "d" and "e" could be operative.
In the DNG files derived from Canon EOS 40D CR2 files, we find "b" and "d"; the value in "b" seems to always be zero. (That is, the black level comes wholly from the per-row "list", "d", known as BlackLevelDeltaV.)
Other bodies
Laurent Clévy, author of an important paper on the CR2 file, reports for a file from the EOS 1D Mark II that the arrangement is what I would describe, in the list above, as "c" only, the values being 1023, 1023, 1023 (for R, G, and B "channels"). (Again, as in my case, this is from analysis of a derived DNG file.)
Best regards,
Doug
**************
Common wisdom has been that in the Canon raw file (.CR2), the sensel values are recorded with a (fixed) offset (1024 in the 14-bit bodies). Thus, the binary value 1024 actually represented a sensel value of 0. The offset here can be regarded as a creature of teh numbering system.
The purpose of doing so is ostensibly to allow for negative sensel values. While such a value would have no meaning photometrically, it can certainly occur in reality as a result of several factors, including variations in the "black level" of the individual sensel and read noise.
This offset value can be thought of as a "black level" for the raw file - a value of that number being considered (for better or worse) to represent "black". (This is not the same thing as the "black point" we may adopt when doing tonal scale optimization.) This is then not so much a creature of the numbering system as a property of the "system".
It now seems as if it is not that simple, at least in the CR2 files of the EOS 40D. Rather, it seems that the Canon CR2 raw file contains a separate black level value for each row of sensels. Ordinarily, these are all in the general region of 1024.
My suspicion is that this is a way to reflect variations in the black levels of individual sensels without determining same for each sensel (and recording that in the file). It may well be that under the way the sensor is read out, the principal causes of black level discrepancy and even read noise may be row-oriented.
Now, I have not actually been able to find this "table" in the CR2 file. I suspect it is held in the MakerNote area in a proprietary format not yet known to me.
But a "tag" (data area) for just thing occurs in the DNG ("digital negative") file - a format developed by Adobe to provide an "industry standard" raw file. If I have Photoshop CS5 convert a Canon CR2 file to a DNG file, and examine the metadata in that file, I find exactly what I describe above. (The tag is called BlackLevelDeltaV - it is part of a broader scheme of recording black levels.)
There is in fact exactly that tag (and its associates) in the current standard for Exif Metadata. I suspect this is a "back formation" from the DNG specification, but I don't know. In any case, the Canon CR2 files do not seem to carry that tag (or the other ones needed in connection with it). Almost certainly, Canon developed their format before these tags were added as bona fide Exif tags.
There are still some mysteries in this area. The raw file analysis program Rawnalyze states, for a Canon CR2 file, the range of black level values in the file in terms of minimum, maximum, and average. For the files I have examined, these values do not agree well at all with the values I fond in the BlackLevelDeltaV tag in the derived DNG file. There could be various explanations for this. Sadly, Gabor Schorr, the developer of Rawnalyze, is no longer with us, so I cannot ask for his help in understanding this.
I am doing further work in this regard.
Of course, there are those who know just how this works, and I have inquired of several of them. I have not yet received any replies. Some colleagues whom I thought might know report that they don't, but would be glad to hear from when when I have found out.
In one typical CR2 file I examined, the range of the 2596 black level values is from 1020.542969 to 1029.398437 (they are carried in 64 bits as a "signed rational"), with an average of 1026.478509.
Some more detail
Here is a little more information on the provision for black levels in the DNG file specification. (It is paralleled in the current Exif metadata specification.)
a. It is possible to state a black level for each sensel.
b. As an alternative it is possible to state a "base" black level pertaining to all sensels regardless of "channel" (R, G, B).
c. As an alternative it is possible to state a "base" black level each channel, applicable to all sensels of the channel.
d. It is then possible to define a "differential" black level by row; this is added to the "base" black level mentioned above for all sensels in the row.
e. It is also possible to define a "differential" black level by column; this is added to the "base" black level mentioned above for all sensels in the column.
f. Potentially, both "d" and "e" could be operative.
In the DNG files derived from Canon EOS 40D CR2 files, we find "b" and "d"; the value in "b" seems to always be zero. (That is, the black level comes wholly from the per-row "list", "d", known as BlackLevelDeltaV.)
Other bodies
Laurent Clévy, author of an important paper on the CR2 file, reports for a file from the EOS 1D Mark II that the arrangement is what I would describe, in the list above, as "c" only, the values being 1023, 1023, 1023 (for R, G, and B "channels"). (Again, as in my case, this is from analysis of a derived DNG file.)
Best regards,
Doug