Open Photography Forums  
HOME FORUMS NEWS FAQ SEARCH

Go Back   Open Photography Forums > Digital Camera Discussion > Canon Eos Mount DSLRs

Reply
 
Thread Tools Display Modes
  #1  
Old August 15th, 2010, 09:38 AM
Doug Kerr Doug Kerr is offline
Senior Member
 
Join Date: May 2006
Location: Alamogordo, New Mexico, USA
Posts: 8,598
Default Offset/black level in Canon raw files

This is to report the conclusion of a long trail of investigation, which has in part been chronicled in another thread.
**************
Common wisdom has been that in the Canon raw file (.CR2), the sensel values are recorded with a (fixed) offset (1024 in the 14-bit bodies). Thus, the binary value 1024 actually represented a sensel value of 0. The offset here can be regarded as a creature of teh numbering system.

The purpose of doing so is ostensibly to allow for negative sensel values. While such a value would have no meaning photometrically, it can certainly occur in reality as a result of several factors, including variations in the "black level" of the individual sensel and read noise.

This offset value can be thought of as a "black level" for the raw file - a value of that number being considered (for better or worse) to represent "black". (This is not the same thing as the "black point" we may adopt when doing tonal scale optimization.) This is then not so much a creature of the numbering system as a property of the "system".

It now seems as if it is not that simple, at least in the CR2 files of the EOS 40D. Rather, it seems that the Canon CR2 raw file contains a separate black level value for each row of sensels. Ordinarily, these are all in the general region of 1024.

My suspicion is that this is a way to reflect variations in the black levels of individual sensels without determining same for each sensel (and recording that in the file). It may well be that under the way the sensor is read out, the principal causes of black level discrepancy and even read noise may be row-oriented.

Now, I have not actually been able to find this "table" in the CR2 file. I suspect it is held in the MakerNote area in a proprietary format not yet known to me.

But a "tag" (data area) for just thing occurs in the DNG ("digital negative") file - a format developed by Adobe to provide an "industry standard" raw file. If I have Photoshop CS5 convert a Canon CR2 file to a DNG file, and examine the metadata in that file, I find exactly what I describe above. (The tag is called BlackLevelDeltaV - it is part of a broader scheme of recording black levels.)

There is in fact exactly that tag (and its associates) in the current standard for Exif Metadata. I suspect this is a "back formation" from the DNG specification, but I don't know. In any case, the Canon CR2 files do not seem to carry that tag (or the other ones needed in connection with it). Almost certainly, Canon developed their format before these tags were added as bona fide Exif tags.

There are still some mysteries in this area. The raw file analysis program Rawnalyze states, for a Canon CR2 file, the range of black level values in the file in terms of minimum, maximum, and average. For the files I have examined, these values do not agree well at all with the values I fond in the BlackLevelDeltaV tag in the derived DNG file. There could be various explanations for this. Sadly, Gabor Schorr, the developer of Rawnalyze, is no longer with us, so I cannot ask for his help in understanding this.

I am doing further work in this regard.

Of course, there are those who know just how this works, and I have inquired of several of them. I have not yet received any replies. Some colleagues whom I thought might know report that they don't, but would be glad to hear from when when I have found out.

In one typical CR2 file I examined, the range of the 2596 black level values is from 1020.542969 to 1029.398437 (they are carried in 64 bits as a "signed rational"), with an average of 1026.478509.

Some more detail

Here is a little more information on the provision for black levels in the DNG file specification. (It is paralleled in the current Exif metadata specification.)

a. It is possible to state a black level for each sensel.
b. As an alternative it is possible to state a "base" black level pertaining to all sensels regardless of "channel" (R, G, B).
c. As an alternative it is possible to state a "base" black level each channel, applicable to all sensels of the channel.
d. It is then possible to define a "differential" black level by row; this is added to the "base" black level mentioned above for all sensels in the row.
e. It is also possible to define a "differential" black level by column; this is added to the "base" black level mentioned above for all sensels in the column.
f. Potentially, both "d" and "e" could be operative.

In the DNG files derived from Canon EOS 40D CR2 files, we find "b" and "d"; the value in "b" seems to always be zero. (That is, the black level comes wholly from the per-row "list", "d", known as BlackLevelDeltaV.)

Other bodies

Laurent Clvy, author of an important paper on the CR2 file, reports for a file from the EOS 1D Mark II that the arrangement is what I would describe, in the list above, as "c" only, the values being 1023, 1023, 1023 (for R, G, and B "channels"). (Again, as in my case, this is from analysis of a derived DNG file.)


Best regards,

Doug
Reply With Quote
  #2  
Old August 15th, 2010, 11:32 AM
Bart_van_der_Wolf Bart_van_der_Wolf is offline
pro member
 
Join Date: Jun 2006
Location: Netherlands
Posts: 4,054
Default

Quote:
Originally Posted by Doug Kerr View Post
This is to report the conclusion of a long trail of investigation, which has in part been chronicled in another thread.
Thanks Doug, for the interesting info.

The fact that not all blacklevel fields are utilized in the DNG, doesn't mean that they are not in the original CR2.

An earlier version of the DNG standard and the conversion tool (if memory serves, before version 2.3 of the DNG converter) totally ignored the black level data and thus were unable to serve as a real alternative for the original Raw file (unless the original was totally embedded in the huge DNG). Later versions did pick this up and thus allowed other/non-Canon Raw converters the possibility to utilize the data, e.g. to reduce banding artifacts.

I suppose all of the Makernote info is incorporated in current DGNs since that earlier omission, but perhaps not all is understood yet (also not by the excellent 'Exif tool' from Phil Harvey).

Cheers,
Bart
Reply With Quote
  #3  
Old August 15th, 2010, 11:46 AM
Doug Kerr Doug Kerr is offline
Senior Member
 
Join Date: May 2006
Location: Alamogordo, New Mexico, USA
Posts: 8,598
Default

Hi, Bart,

Quote:
Originally Posted by Bart_van_der_Wolf View Post
The fact that not all blacklevel fields are utilized in the DNG, doesn't mean that they are not in the original CR2.
Indeed.

I hope to learn more from Phil Breeze when he gets back from vacation, and perhaps from Dave Coffin as well.

Quote:
An earlier version of the DNG standard and the conversion tool (if memory serves, before version 2.3 of the DNG converter) totally ignored the black level data and thus were unable to serve as a real alternative for the original Raw file (unless the original was totally embedded in the huge DNG). Later versions did pick this up and thus allowed other/non-Canon Raw converters the possibility to utilize the data, e.g. to reduce banding artifacts.

I suppose all of the Makernote info is incorporated in current DGNs since that earlier omission, but perhaps not all is understood yet (also not by the excellent 'Exif tool' from Phil Harvey).
DNG's blown by PS CS5 from 40D CR2's do include the Makernote (although I can't be sure in its entirety). But Exiftool does not seem to understand the black level stuff there.

I am working with Phil Harvey on this. But so far most of our discussion has been on helping me to drive Exiftool from the command line (it's easier said than done).

As you can imagine, it has been a helluva journey up to this point!

Thanks for your inputs.

Best regards,

Doug
Reply With Quote
  #4  
Old August 16th, 2010, 08:48 AM
Doug Kerr Doug Kerr is offline
Senior Member
 
Join Date: May 2006
Location: Alamogordo, New Mexico, USA
Posts: 8,598
Default

Well, it seems that I was not at the end of the story.

By way of review

The DNG file format provides for black levels:

a. By individual sensel, or
b. On an global basis (all sensels)*, or
c. On a "channel" basis (for all sensels of a channel: R, G1, G2, B), in a list for all channels*

* Optionally, these can be then incremented by either of these or both:

d. An increment pertaining to all sensels in a column (in a table)
e. An increment pertaining to all sensels in a row (in a table)

My earlier conclusion

Examination with ExifTool of DNG files blown by Photoshop from Canon EOS 40D CR2 raw files suggests that the scheme there is "b" plus "e". (Thus is what I reported at the head of this thread.)

I have tables of the actual "e" values (plus the "b" value), from the DNG tags, for a number of DNG files derived from Canon EOS 40D Cr2 files. The "b" value is always zero; the "e" values are in the general range 1021-1028. (They are recorded to a precision of 1/256 unit.)

Moving on

The raw data analysis program Rawnalyze reports, for the file in place, the maximum, minimum, and average of the black level values it will apply. A nagging concern has been that the range reported is always considerablyh wider than the range of "e" values ExifTool reads from the DNG file, on both the low and high ends. (The average is almost always 1024.)

Rawnalyze does not provide for illustrating the black level values, by sensel, on a graphic basis (such as by letting us set a threshold that will allow all sensels whose black level value is greater than that to "light up").

I can determine the applied black level correction for any specific sensel by observing the value of the sensel with black level correction turned on and then with it turned off. But I hardly want to do a survey of this for the 10M sensels in these files!

But this morning I made two fascinating discoveries. My test piece was an EOS 40D CR2 raw file that was "wholly substantially blown out"; that is, the frame has been overexposed to perhaps 10 stops beyond metered exposure.

I used the Rawnalyze "threshold" display to visually examine the distribution of recorded sensel values. The preponderance of them were at 13828, but some were lower. The lowest were at 13816 (12 units lower).

Amazingly, for any value, all the sensels having it were in contiguous rows. That is, I might find, that for value 13816, one row and one row only containing sensels at that value, and every sensel in the row had that value. At value 13820, there would be a number of rows having sensels of that value, in each case every sensel in the row having that value.

We see this here (all sensels with values less than 13823 "lit"; only a small area shown, but the lines extend across the entire frame):



This presumably reflects a variation in behavior in the original sensor readout on a row basis, probably related to the use of certain common circuitry for each row.

Next, I enabled black level correction (so that Rawnalyze would subtract from the stored value for each sensel whatever black level it had learned, somehow, should be applied to that sensel).

To skip for a moment the exciting part, I noted that now the range of corrected values for the file I discuss was from 12794 through 12813, a range of 19 units. This of course bespeaks a black level correction, in all cases, in teh general area of 1024, as we might expect.

But the exciting part was this. As I adjusted the threshold so that only (adjusted) sensel values above a certain level would "light up", there was a clear "crosshatch" pattern. You an see an example here (only the "red" channel pixels are enabled here, for clarity):



Since there was no "column" pattern in the recorded (uncorrected) sensel values, this clearly shows that there must be a "column" pattern (as well as a "row" pattern) in the applied black levels.

This then suggests that the black level correction applied by Rawnalyze has both "d" and "e" aspects.

The further conclusion is that:

Both "d" and "e" tables are in the CR2 file (I don't know where, nor does ExifTool); doubtless someplace in the MakerNote.
Photoshop CS4 only transcribes the "e" table into the corresponding tag in a DNG file, not the "d" table.
Rawnalyze, working with a Canon-origin DNG file, uses the original Canon black level information rather than its transcription into DNG tags. (Photoshop preserves the MakerNote data when converting a CR2 file to a DNG file. It did not always do that, as Bart van der Wolf has pointed out.)

I will continue to poke around, perhaps killing the MakerNote value in test files with ExifTool and seeing what Rawnalyze does with that.

Best regards,

Doug
Reply With Quote
  #5  
Old August 16th, 2010, 09:39 AM
Doug Kerr Doug Kerr is offline
Senior Member
 
Join Date: May 2006
Location: Alamogordo, New Mexico, USA
Posts: 8,598
Default

A simplistic (and not at all definitive) test suggests that the black level information in a Canon CR2 raw file may not be held in the MakerNote area of the Exif metadata, as I had conjectured.

I took a copy of such a file and asked ExifTool(GUI) to delete the MakerNote data.

Having done so, Rawlanyze's outlook on black level correction was apparently unchanged.

Best regards,

Doug
Reply With Quote
  #6  
Old August 16th, 2010, 10:44 AM
Bart_van_der_Wolf Bart_van_der_Wolf is offline
pro member
 
Join Date: Jun 2006
Location: Netherlands
Posts: 4,054
Default

Quote:
Originally Posted by Doug Kerr View Post
A simplistic (and not at all definitive) test suggests that the black level information in a Canon CR2 raw file may not be held in the MakerNote area of the Exif metadata, as I had conjectured.

I took a copy of such a file and asked ExifTool(GUI) to delete the MakerNote data.

Having done so, Rawlanyze's outlook on black level correction was apparently unchanged.
Hi Doug,

I remember reading, during the development cycle of different versions, that certain stuff about the blackpoint per camera model is hardcoded in the Rawnalyze logic. It was mentioned in those "what's new" kind of remarks when new models were added to the application's detection logic. So maybe it is in the Makernotes, but Rawnalyze doesn't rely on it (perhaps for encoding purposes or to avoid missing data to spoil the fun).

Cheers,
Bart
Reply With Quote
  #7  
Old August 16th, 2010, 11:09 AM
Doug Kerr Doug Kerr is offline
Senior Member
 
Join Date: May 2006
Location: Alamogordo, New Mexico, USA
Posts: 8,598
Default

Hi, Bart,

Quote:
Originally Posted by Bart_van_der_Wolf View Post
Hi Doug,

I remember reading, during the development cycle of different versions, that certain stuff about the blackpoint per camera model is hardcoded in the Rawnalyze logic.
Probably black level is a better term for the quantity of interest. Black point is pertinent to the interpretation of the tonal scale. In any case, here I follow the distinction as used in the Rawnalyze documentation. You will see an example of that below.
I think that does not pertain here - the range and average of black point values cited varies from file-to-file (even for the same camera model and copy).

The Rawnalyze documentation says:
The initial value of the black point is either

* the minimum of the black level correction values, if there are any; these are gained either from the descriptive data of the raw file, or calculated [from] the so-called masked pixels around the image [or]

* 0 if black level correction does not apply.
There is definitely camera-specific information hard coded intro Rawnalyze. The documentation specifically speaks of this with regard to clipping levels and the range to be used (on the DN axis) for the histogram (related to the matter of White Point).

All very interesting.

Thanks for your continuing inputs.

Best regards,

Doug
Reply With Quote
  #8  
Old August 18th, 2010, 10:56 AM
Doug Kerr Doug Kerr is offline
Senior Member
 
Join Date: May 2006
Location: Alamogordo, New Mexico, USA
Posts: 8,598
Default

Over the last couple of days, further study, plus some input from others, has brought me to a new outlook on the matter of black level correction in Canon EOS CR2 raw files. Important input came from Dave Coffin, developer of DCRAW, an important open-source raw conversion engine used in many applications.

There are still some things that don't fit together; I hope to resolve these soon.

Maybe this is the answer

Black level information is not "stored" in a Canon CR2 file (in the sense of being in a "table" or a metadata tag).

In the "grid" of sensel output values in a CR2 file, six columns are from sensels that are "masked" from light, three at each side of the "main grid" of values from "working" sensels.

Note that, in an "odd-numbered" row, the six "masked" pixels are alternately from the "R" and "G1" positions ("channels") in the CFA pattern; in an "even-numbered" row, alternately from the "G2" and "B" positions.

DCRAW examines the values in the masked columns immediately adjacent to the main grid. (Coffin says that the outboard ones often have anomalous values).

In an odd-numbered row (having "R" and G1" sensels), the masked sensel just to the left of the main grid is a "G1" sensel, the one just to the right an "R" sensel.

DCRAW examines uses the value of the masked sensel of a particular flavor as the black level for all the sensels of the corresponding flavor in that row. That is, in an odd-numbered row (having "R" and "G1" sensels), the value of the masked sensel just to the left of the main grid is used as the black level for all the "G1" pixels in that row; the value of the masked sensel just to the right of the main grid is used as the black level for all the "R" pixels in that row.

In an even-numbered row, the story is the same, except that the players are "G2" and "B" flavor sensels.

In fact, the top and bottom rows of sensel values are also from "masked" sensels. DCRAW does not examine these with respect to black level matters. (We might imagine a "row and column" schema.) Coffin says they "often have strange patterns" and thus cannot be used reliably in such a schema.

In DNG files

Note that the DNG specification says (to paraphrase):
Even if the masked sensels are included in the raw data, with the intent that the receiving application determine the black levels using some favorite algorithm, the resulting black levels should also be stored in the file so that a receiving application that does not want to use some favorite algorithm of its own can just use the values already determined by the "generating" application.
Note however that the "table" form provided in DNG (in two metadata tags) does not provide for separate black levels for all the "R" and "G1 sensels of an odd row, or all the "G2" and "B" sensels of an even row. (Differences between the black levels for "R" and G1" sensels and between "G2" and "B" sensels can only be stated on a "global" basis.) DNG does support both "row-wise" and "column-wide" tables, both of which may appear.

The Adobe engine

Evidently the Adobe raw conversion engine takes the same view as DCRAW (perhaps it uses that engine): develop only row-wise black level values; the clue is that a DNG file generated by the Adobe engine includes black levels (in a "table") but only in the row-wise aspect.

In contrast, it seems as if Rawnalyze (which may well not use the DCRAW engine) develops both row-wise and column-wise black level components.

Acknowledgment

My thinks to the many corespondents who have weighed in on this, even those who said, "I have no idea, but let me know if you find out".

Special thanks to Dave Coffin, the one guy I knew would know the answer.

Finally, of course, the greatest thanks to my wife Carla, who understood and supported my obsession with cracking this mystery.


Best regards,

Doug

Last edited by Doug Kerr; August 18th, 2010 at 02:24 PM.
Reply With Quote
  #9  
Old August 18th, 2010, 02:07 PM
StuartRae StuartRae is offline
Senior Member
 
Join Date: Jul 2006
Location: Wiltshire, UK
Posts: 1,025
Default

Hi Doug,

Thank you so much for your dogged determination to arrive at the truth. What you say now ties in with the DNG spec. from Adobe, although they (as far as I can tell) make no distinction between odd and even rows.
Even though this information is of no direct use to me, it's always nice to know how things work.

Quote:
"odd-numbered" row, the six "masked" pixels are alternately from the "R" and "G1" positions ("channels") in the CFA pattern; in an "even-numbered" row, alternately from the "G1" and "B" positions.
You may wish to correct this, assuming I've correctly understood what follows. I assume you mean "even-numbered" row, alternately from the "G2" and "B" positions.

Regards,

Stuart
__________________
-----------------------------------
C&C and edits always welcome
Reply With Quote
  #10  
Old August 18th, 2010, 02:23 PM
Doug Kerr Doug Kerr is offline
Senior Member
 
Join Date: May 2006
Location: Alamogordo, New Mexico, USA
Posts: 8,598
Default

Hi, Stuart,

Quote:
Originally Posted by StuartRae View Post
Thank you so much for your dogged determination to arrive at the truth. What you say now ties in with the DNG spec. from Adobe, although they (as far as I can tell) make no distinction between odd and even rows.
No, nor between the two flavors of sensels on one row (e.g., R vs. G1, G2 vs B) either.

Quote:
Even though this information is of no direct use to me, it's always nice to know how things work.
Sure, no use to me right now either. But you never know . . .


Quote:
You may wish to correct this, assuming I've correctly understood what follows. I assume you mean "even-numbered" row, alternately from the "G2" and "B" positions.
Thanks for the catch. All that stuff had been copied and pasted, inverted and reversed, until it was like over-kneaded dough. I'm surprised that any of it ended up rightside up!

I just made a "silent" correction.

Thanks for writing.

Best regards,

Doug
Reply With Quote
  #11  
Old August 18th, 2010, 05:55 PM
Doug Kerr Doug Kerr is offline
Senior Member
 
Join Date: May 2006
Location: Alamogordo, New Mexico, USA
Posts: 8,598
Default

Here is a diagram I made that shows the allocation of the real estate on a Canon EOS 40D sensor as represented in a CR2 raw file.

The finer details came in part from information provided by Dave Coffin.



The figure is at a rather large scale so as to present the detail in the "border" regions. Since the entire figure would then be enormous, large vertical and horizontal regions have been "snipped" (and are represented by the crosshatched areas in the figure). As a result, the "border areas" look disproportionately large.

I do not know what goes on in the top 18 sensel rows or the leftmost 30 sensel columns of the sensor.

Note that locations are given in terms of "coordinates" of boundaries between sensel cell rows or columns, not in terms of sensor row or column counts (which can lead to irritating 1 errors in reckoning - the so-called "fencepost" errors).

"Raw field" refers to the realm of the entire "grid of sensel values" in the CR2 file.

"Masked sensels" are the locations of the outputs from sensels that are masked (kept in the dark during exposure). They are the basis for the development of black level data, as discussed in my earlier message.

"Active area" refers to the suite of sensel values that will be an input to demosaiaing.

"Image area boundary" refers to the geometric region that corresponds to the developed image; there are no "image pixels" in the raw file.

Information in the metadata of the CR2 file paints a slightly different picture, illustrated here:



The difference is in the disposition of the masked sensels. My guess that this is the result of a "misunderstanding" by the Adobe raw engine in generating the metadata. The whole area is very complicated, however, so there may be some other significance to the discrepancy.

Best regards,

Doug
Reply With Quote
  #12  
Old December 6th, 2010, 09:50 AM
Peter Galbavy Peter Galbavy is offline
Member
 
Join Date: Jun 2006
Location: London, UK
Posts: 66
Default

Sorry to wake an old thread...

I suspect that the masked sensels around the frame also contribute to the thermal noise reduction algorithms applied, especially at high ISO and long exposure. If you do not take a "black frame" (an option on the later Canon's but mandatory for long exposures on earlier bodies, from memory) for subtraction then how else will the firmware or RAW converter be able to guess what the noise level might be ?
Reply With Quote
Reply

Bookmarks

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are Off
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
The "offset" in Canon raw files Doug Kerr Imaging Technology: Theory, Alternatives, Practice and Advances. 9 August 14th, 2010 10:28 PM
Zeiss glass on Canon dSLRs John_Nevill Lenses: DSLR and Rangefinder, MF adaptions to 35mm such Zoerk 9 May 22nd, 2010 12:53 AM
News: Canon announces new 500D Ivan Garcia Canon Eos Mount DSLRs 10 May 1st, 2009 11:55 PM
FS, UK various Canon Items Ivan Garcia Buy and Sell Photo Equipment: Excess gear by participating members. Pictures please! 1 September 23rd, 2008 05:49 PM
Desire to write a new, free raw converter Daniel_Hyams Image Processing and Workflow 13 August 1st, 2006 02:43 PM


All times are GMT -7. The time now is 09:37 PM.


Posting images or text grants license to OPF, yet of such remain with its creator. Still, all assembled discussion 2006-2017 Asher Kelman (all rights reserved) Posts with new theme or unusual image might be moved/copied to a new thread!