Canon 1D3 dynamic range test results

Hi All,
My friend got a new Canon 1D3 and we were curious to see how it compared to the Leica DMR which we both have. I have posted infrequently here at OPF since having my second child, but figured this information might be useful to people considering the new canon body. I was intrigued by the higher bit processing of the 1D3 and wondered if it might be good enough to cause me to replace my Leica DMR (digital back).

Test Method: For the test pictures were take of a Stouffer t41 transmissive film mounted on a lightbox using both bodies and the same lens (leica 100mm macro) on both cameras. Exposure was carefully set to get the lightest setting to register 250-255 in the computer. Lightroom 1.1 was used to convert the raw files and crop the images but no adjustments were made. Imatest Pro software was used to analyze the data and generate the reports shown below.

Results: You can see from the charts that the 1d3 edged out the DMR in total DR by a hair 12.7 to 12.6 stops, but the really useful numbers are the low, med, and high where the DMR still has roughly a 1.5 stop edge over the new camera.

Canon 1D3



Leica DMR




Conclusion: For now, I am going to keep my DMR
Regards,
Eric Hiss

Hi Eric,

This is very interesting data.

I like the approach and the result is impressive for the Leica.

Still the 3 graphs are packed with data. Could you take each pair,

• Density response
  
• Noise (Luminence)
  
• Noise normalized

A lot of photographers are not as yet familiar with the terms used so this would be especially helpful, dealing with the Imatest, a major impartial tool and these two important cameras.

I wish that the DMR was still being sold!

Asher

Interesting comparison

Here is something that has always bothered me about these kinds of tests;

Eric Hiss said:

Hi All,
Lightroom 1.1 was used to convert the raw files and crop the images but no adjustments were made.
Eric Hiss

Click to expand...

"No adjustments"? Lightroom is an adjustment as is Camera RAW, Capture One or Canon Standard Picture Style. I always wonder what the test results would have been if some other form of non-adjustment had been used. If comparing 1D3 against 1D3 how would the same image in Landscape compare against itself in Neutral?

Jack Joseph Jr said:

H If comparing 1D3 against 1D3 how would the same image in Landscape compare against itself in Neutral?

Click to expand...

Jack, do you mean 1DII against 1DIII?

Asher

Jack Joseph Jr said:

If comparing 1D3 against 1D3 how would the same image in Landscape compare against itself in Neutral?

Click to expand...

One might be a much better camera

Asher Kelman said:

Jack, do you mean 1DII against 1DIII?

Asher

Click to expand...

The point is that the conversion parameters dictate the DR of the output file.

Conversions throw away detail all the time. What they throw out depends on the settings.

I've measured the actual ISO 100 RAW DR of the 1Dmk3, and it is almost exactly the same as the 1Dmk2, with the conventional definition of max signal divided by blackframe noise:

mk2: 3583/1.26 = 2843

mk3: 14256/4.88 = 2921

EDIT:Sorry. 15280 in my OP was the value before I subtracted the blackpoint. The real value is 14256. The result was correct.

log(2921/2843)/log(2) = 0.039 stops more DR for the mk3, unless of course, the samples I had were from an excellent mk2 and a poor mk3.

Hi Asher,
The imatest website (www.imatest.com) has quite a bit of information on it and probably much better explanations on it than I could do. btw- I would guess that the M8 would turn in very similar results.

John,
Yes I believe that the numbers are subjective to what kind of RAW software is used as well as what firmware is in the camera. Just as point of reference, the 1D3 sample image when converted in c1 had a much lower total number but luckily lightroom came out with an update at just the right time and I was able to convert it with that which produced a higher number. I suspect that this has a lot to do with black and white point settings. I think C1 sets this in some automatically just as lightroom starts with the shadow point at 5 by default. It sounds like you are very knowledgeable in this subject - do the numbers seem accurate to you?

Eric

No, 1D3 vrs 1D3

Asher Kelman said:

Jack, do you mean 1DII against 1DIII?

Asher

Click to expand...

Since the OP converted the image being tested using LR, e.i. Adobe's Picture Style so to speak I'm curious how the comparison tests would look if the exact same image taken by the same camera was converted in using two different schemes. I used Canon Standard against Neutral as an example since the later seems to preserve more highs. Another pair to compare could be ACR factory default against ACR Auto. It seems like were talking about conversions here rather than actual camera capabilities. I stand corrected if I'm wrong here.

Eric Hiss said:

Hi Asher,
I think C1 sets this in some automatically just as lightroom starts with the shadow point at 5 by default.

Click to expand...

Hi Eric,

C1 does not set the bp or wp automatically unless you click the auto-exposure button. I'm not familiar with how Imatest works but I suspect the differences you're seeing between converters is due to default tonality settings - default ACR probably yielding a flatter response curve than C1. In C1 you might try selecting the film curve "linear response" and see what you get.

- DL

John Sheehy said:

The point is that the conversion parameters dictate the DR of the output file.

Conversions throw away detail all the time. What they throw out depends on the settings.

Click to expand...

Absolutely. It would have been interesting to see Imatest's verdict on a linear gamma conversion e.g. in DPP, although that would sort of invalidate a direct comparison with the Leica DMR.

The current C1 3.77 only offers a preliminary Raw conversion functionality for the 1D Mark III, so that would make a C1 based comparison iffy at best. Lightroom is supposed to have a more mature Raw conversion for the Mark III, but it also is relatively new. In Lightroom one should disable the blackpoint correction and use a linear tone-curve to reduce the mangling of raw data. The density curves suggest that a non-linear tone curve was used for both cameras! Even White-balancing should be avoided as much as possible, so a default daylight balance would probably be the least intrusive (white balancing a tungsten? lightbox would result in serious boosting of the blue/green channels).

Anyway, the Imatest evaluations show a higher DR for the Mk III than any previous Canon model (they usually stopped at 11 stops and then some), so that's encouraging.

Bart

John Sheehy said:

I've measured the actual ISO 100 RAW DR of the 1Dmk3, and it is almost exactly the same as the 1Dmk2, with the conventional definition of max signal divided by blackframe noise:

mk2: 3583/1.26 = 2843

mk3: 14256/4.88 = 2921

Click to expand...

The blackframe noise, assuming the same ISO setting and exposure time, seems suspiciously high for the Mk3. Are you sure there is no error involved? On the other hand, for a 16-bit output file the Mk2 noise will be multiplied by 16 while the Mk3 will be multiplied by 4. But that only pulls them to approx. the same level in output, contrary to what many sample images show.

Puzzling.

Bart

Eric Hiss said:

I was intrigued by the higher bit processing of the 1D3 and wondered if it might be good enough to cause me to replace my Leica DMR (digital back).

Click to expand...

Hi Eric, thanks for sharing the results of your evaluation of two fine cameras. Would you mind sharing the Mark III Raw, e.g. through Yousendit or a temporary download?

Thanks, anyway.

Bart

Bart_van_der_Wolf said:

Even White-balancing should be avoided as much as possible, so a default daylight balance would probably be the least intrusive (white balancing a tungsten? lightbox would result in serious boosting of the blue/green channels).

Click to expand...

WB has a tremendous effect on DR, especially when considering what different converters do with the non-overlapping ranges. If your subject is grey/white, then it will have the same response in all 3 color channels, only if the light source is a certain purplish-magenta. Even direct sunlight results in the red channel being a stop weaker than the green, and the blue somewhere in-between. That means that if you have 12 stops of DR in one channel, then the red and green will be shifted by 1 stop, with the green getting a good stop of shadows that the red missed, and the red getting a good stop of highlights clipped by the green. The DR of full color is therefore only 11 stops, while the DR of a conversion that renders greyscale deep shadows only from the green channel (don't know if any available software works this way, but such an option would be nice), and the red channel is used for highlights clipped in the green channel, for a total of 13 stops.

Tungsten is up to 2 stops weaker in the blue channel than the red and green (which are almost equal with the added reds); that means two stops of full-color DR missing for 10 stops, but a potential of 14 stops with two extra with greyscale shadows and highlights.

Bart_van_der_Wolf said:

The blackframe noise, assuming the same ISO setting and exposure time, seems suspiciously high for the Mk3. Are you sure there is no error involved? On the other hand, for a 16-bit output file the Mk2 noise will be multiplied by 16 while the Mk3 will be multiplied by 4. But that only pulls them to approx. the same level in output, contrary to what many sample images show.

Puzzling.

Click to expand...

This is ISO 100. The noise improvements are only supposed to be at the high end of the ISO scale.

My values are standard deviations, however, and don't necessarily tell the whole story. It is quite possible that the mk3 has less 1-dimensional banding noise at ISO 100, as it seems to do at ISO 1600 and 3200, and this really doesn't show up in standard deviations. You need to average entire lines or columns of pixels to measure their strength (not just crops, either, as the small samples of crops are going to have bigger differences in mean line and column values).

It is also possible that even though the extra 2 bits really doesn't improve RAW data much directly, it may force software (or firmware) to use more bits of precision, giving less posterization in the final results. Quantization is a more serious a matter in post processing than it is in actual RAW data, IME. A few steps of arithmetic at low precision can decimate RAW data far more than a missing bit or two in the RAW data, especially at high ISOs.

Regardless of the numbers, the mk3 looks good at all ISOs.

Hopefully, the successor to the 5D will have these banding and high ISO improvements, too, and I will probably buy one. Ditto for the 30D successor. Canon has never held back on high-ISO quality in the mid-range cameras before; only in the Rebel series.

I'll try and put the raw files up

Hi Bart,
I'll put the raw files from both cameras available for a short while from my website. Then others that want to try different RAW converters or methods may do so. Link to directory is here:
www.eh21.com/TestFiles/

If you do download the test files, please send me a jpg of the cropped chart (full size) and I will be happy to run it through Imatest again to see if the results differ. Make sure to tell me what RAW conversion method you used.

E-mail Artist @ eh21.com (take out the blanks)

Eric Hiss said:

Hi Bart,
I'll put the raw files from both cameras available for a short while from my website. Then others that want to try different RAW converters or methods may do so. Link to directory is here:
www.eh21.com/TestFiles/

Click to expand...

The brightest patch in the cr2 file is 0.55 stops below the 1Dmk3 ISO 100 RAW clipping point.

Lot's of good points brought up:

I didn't use a linear curve for either of the conversions nor did we set the white balance on the cameras. The conversions were the same for both cameras however. Getting exposure correctly was difficult because the cameras only have 1/2 or 1/3 stop settings and we wanted to get the lightest patch to read between 250 and 255. The program throws a warning if exposure is off more than that.

I think it does do some correction for both tone curves, and white balance but I could be wrong. At least in the instructions these are not called out as important things to set. I'll write the author of the software, Norman Koren, and ask him.

Eric Hiss said:

Hi Bart,
I'll put the raw files from both cameras available for a short while from my website. Then others that want to try different RAW converters or methods may do so. Link to directory is here:
www.eh21.com/TestFiles/

If you do download the test files, please send me a jpg of the cropped chart (full size) and I will be happy to run it through Imatest again to see if the results differ. Make sure to tell me what RAW conversion method you used.

Click to expand...

Hi Eric, thanks again.

I ran various conversions through Imatest (same version as you used) myself, and also came to the conclusion that the CR2 file is approx. half a stop under-exposed. Canon's DPP also allows to increase "Exposure" by 0.4+ before clipping.

I also came to the conclusion that for the step detection method used by Imatest, a (simple) gamma adjusted file is just fine. It does help to have the stepwedge as horizontal as possible, that helps the step detection.
So a conversion to Adobe RGB with Lightroom 1.1 can be used, but do disable all the "Sharpening" controls (that automatically includes noise reduction), and use a linear tonecurve with Recovery, Shadows and Brightness all set to '0'. Different White-balances in Lightroom do not seem to make too much of a difference, although there may be between different applications.

As said, all Raw converters alter the data and they take different trade-offs between noise and sharpness/detail. So as to avoid as many Raw conversion variables as possible I tried accessing the Monochrome Bayer data with ImagesPlus. There seems to be some teething problem with its Mark III Raw conversions in the latest beta version I'm testing, but I may have a resolution for that by the end of next week. I'll keep you posted about the results.

If you (or your friend) still have bracketed versions of your sequence, probably +1/3rd and +2/3rd relative to the version you used, then they would be welcome. The result might even persuade me to get a Mark III myself ...

Bart

I think it would be instructive to also include the Fuki S5 in this test since it can get close to a DR of 12 apprarently.

This is where the Fuji would be valuable in wdding photography for getting suits and the detail of white wedding gowns covered well.

Asher

Asher Kelman said:

I think it would be instructive to also include the Fuki S5 in this test since it can get close to a DR of 12 apprarently.

This is where the Fuji would be valuable in wdding photography for getting suits and the detail of white wedding gowns covered well.

Click to expand...

That wouldn't be a problem for the Imatest software that Eric used and I use it also. However it would need someone to shoot a transparent stepwedge (Stouffer 4110 or similar) with the Fuji S5 camera.

I would be happy to do the Imatest evaluation, and post a new topic in the appropriate forum.

Bart

Hi Bart,

What influence does it have on Imatest results if one gets the Stouffer 4100 or the more expensive calibrated and calibratd and certified versions?

Is there ability of Imatest to benefit from the Calibration or C&C? Presumably, all camera would have to be done with the same wedge?

Is there means within Imatest to accomodate a calibration data set?

Asher

Asher Kelman said:

Hi Bart,

What influence does it have on Imatest results if one gets the Stouffer 4100 or the more expensive calibrated and calibratd and certified versions?

Click to expand...

There is little benefit for Imatest, there can be more of a benefit for laboratory use where calibration to known luminance is essential. Imatest is less interested in the exact density as it is in the density increment, for the fitting of a gamma curve and determining the threshold step differences versus noise. The uncalibrated Stouffer wedges are pretty decent in their densities and increment anyway, they just don't come with calibration value on paper.

Is there ability of Imatest to benefit from the Calibration or C&C? Presumably, all camera would have to be done with the same wedge?

Click to expand...

Imatest takes a lot of internal precautions to make results as repeatable as possible, but the most important step is the taking of the image. The backlighting must be uniform, and one must avoid stray light reaching the front of the target (reduces D-min values most) and the lens (to avoid flare and internal reflections as much as possible). Errors in this setup will usualy outweigh small variations in the target. Also remember that the 4110 step wedge has 0.10 density increments (=1/3rd stops) and that's about as accurate as we can expose anyway.

Is there means within Imatest to accomodate a calibration data set?

Click to expand...

Not with the module for step wedge evaluation, although one can output a file with the measured results and import that e.g. in Excel where the calibrated input data can be added, but again it will only have a marginal effect on the outcome. There are on the other hand alternative methods that are more accurate, but they are quite labor intensive and require many input files and postprocessing/evaluating.

Bart

Hi Bart,
Thanks for your work here. As far as having another exposure for the 1D3 - sadly no. We knew it was a bit under, but on C1 it was only .15 stops under (another mystery). Because we used the stouffer transmissive step chart over a light box, there was no way to modulate the light, and the camera as you know only does 1/3 stops. We did bracket up and down in 1/3 stops thinking the most we could be off is 1/3 stop. After taking the images we reviewed in C1 and took the image that was closest to 255 without going over. How could it be less than 1/3 stop under? Can this be a function of input profiles C1 uses? I certainly wonder as there were some big differences between C1 and lightroom in the data.

For the purposes of imatest the native raw file needs to read 250-255 which I think it does or if not 249 so we felt this was pretty close. I did not see anything different in lightroom so I am surprised to hear you saw that it was off by more.

As an aside, the 1D3 has the Highlight priority feature which we did not use or test.

Bart_van_der_Wolf said:

Hi Eric, thanks again.

As said, all Raw converters alter the data and they take different trade-offs between noise and sharpness/detail. So as to avoid as many Raw conversion variables as possible I tried accessing the Monochrome Bayer data with ImagesPlus. There seems to be some teething problem with its Mark III Raw conversions in the latest beta version I'm testing, but I may have a resolution for that by the end of next week. I'll keep you posted about the results.

Click to expand...

IRIS (http://www.astrosurf.com/buil) can load the 1Dmk3 files, but it really has no specific knowledge of the camera, just of the Canon CR2 compression scheme, so it spits out the full RAW image, with the non-exposed areas and all (not that I would complain about this; I wish the option were available for all RAWs that had this extra "image" material), but you can crop this in IRIS (or more easily in another program). You can split the CFA image into 4 color planes with the command "split_cfa g2 b r g1" (that's the pattern for Canons) and it will keep the g2 (one of the green) channels in memory, and you can load any of the others, like red, with the command, "load r".

You can output .pic or .fit from IRIS, which I believe ImagesPlus can open at least one of.

If you try to test the entire CFA image, it will have lots of noise in a checker pattern due to the different exposures of the color channels.

Eric Hiss said:

Hi Bart,
Thanks for your work here. As far as having another exposure for the 1D3 - sadly no.

Click to expand...

A pitty, but such is life.

We knew it was a bit under, but on C1 it was only .15 stops under (another mystery). Because we used the stouffer transmissive step chart over a light box, there was no way to modulate the light, and the camera as you know only does 1/3 stops. We did bracket up and down in 1/3 stops thinking the most we could be off is 1/3 stop. After taking the images we reviewed in C1 and took the image that was closest to 255 without going over. How could it be less than 1/3 stop under?

Click to expand...

In principle that's true, but in practice it may be beneficial to over-expose 1/3rd of a stop. While that will clip the step numbered 1, the number 2 step may be closer to the 1/3rd stop exposure accuracy! The 4.05 net density range of the 4110 step tablet offers some 13.45 stops of range, so we can lose 1/3rd of a stop without losing too much of the 14th bit of the ADC (which is mostly noise and ADC quantization errors anyway), or capture the Raw converter profile differences.

Can this be a function of input profiles C1 uses? I certainly wonder as there were some big differences between C1 and lightroom in the data.

Click to expand...

Yes. Having made several custom made profiles myself, I can say that one can define the profile to render the luminance brighter or darker. The profile also depends on the signal levels of the test target as recorded during exposure (also within 1/3rd stop accuracy!).

For the purposes of imatest the native raw file needs to read 250-255 which I think it does or if not 249 so we felt this was pretty close. I did not see anything different in lightroom so I am surprised to hear you saw that it was off by more.

Click to expand...

You are correct about the Lightroom interpretation, but the Canon DPP interpretation shows more than 1/3rd of a stop under-exposure (which is also close to what John Sheehy got from IRIS).

As an aside, the 1D3 has the Highlight priority feature which we did not use or test

Click to expand...

Correct, and one shouldn't use it for actual DR measurements, but it is interesting to test whether its effects propagate to the Raw data or if it is just a tonecurve preset (combined with an ISO shift, which I presume). In fact one should also not use the long exposure noise reduction (black frame subtraction) feature as it might kick in unexpectedly, and one should shoot with an aperture of approx. 5.6 or a bit smaller to avoid lens vignetting when shooting the target.

Bart

John Sheehy said:

IRIS (http://www.astrosurf.com/buil) can load the 1Dmk3 files, but it really has no specific knowledge of the camera, just of the Canon CR2 compression scheme, so it spits out the full RAW image, with the non-exposed areas and all

Click to expand...

Yes, IRIS is a capable alternative, the price certainly is hard to beat ;-), but I have the feeling that it is too much based on the DCRAw code (it used to be anyway). That included a black-point interpretation that may be not exactly what the Raw file data offers. Maybe it has changed since I last evaluated it (DCRaw itself has also evolved).

You can output .pic or .fit from IRIS, which I believe ImagesPlus can open at least one of.

Click to expand...

Yes, FITS is a common file format in astronomy.

If you try to test the entire CFA image, it will have lots of noise in a checker pattern due to the different exposures of the color channels.

Click to expand...

Yes, that's another (workflow) difference between IRIS and ImagesPlus. IP allows to split, without any(!) processing, in a Red / Green / Blue channel file before Bayer demosaicing and without interpolation, just the pure Raw per channel data which is only scaled for 16-bit/channel output (e.g. TIFF), which allows for efficient per channel statistical analysis.

We'll see what the final differences of the same file tell us.

Bart

Bart_van_der_Wolf said:

Yes, IRIS is a capable alternative, the price certainly is hard to beat ;-), but I have the feeling that it is too much based on the DCRAw code (it used to be anyway). That included a black-point interpretation that may be not exactly what the Raw file data offers. Maybe it has changed since I last evaluated it (DCRaw itself has also evolved).

Click to expand...

IRIS only uses the DCRAW code to decompress the RAW data. IRIS loads RAW files with the exact RAW data in the files. It pays no attention to blackpoint. You have to mind it yourself.

Unless you have a long exposure, or your camera is defective, the blackpoint is 128 for current 1D*, 5D, 20D, and 30D cameras, and 256 for the rebels. The blackpoint may float almost 1 ADU higher for high ISOs, from the tiny bit of dark current present even in normal shutter speeds.

You also need to do a completely blown-out exposure at the same ISO to determine what the RAW clipping point is. It is only 4095 on the 20D, and with ISO 3200 on other cameras, and with some ISOs (like multiples of 125 on the 30D). It is around 3650 to 3720 on the 1D* and 5D cameras. Canon does not like to use all 4095 levels.

Yes, FITS is a common file format in astronomy.

Yes, that's another (workflow) difference between IRIS and ImagesPlus. IP allows to split, without any(!) processing, in a Red / Green / Blue channel file before Bayer demosaicing and without interpolation, just the pure Raw per channel data which is only scaled for 16-bit/channel output (e.g. TIFF), which allows for efficient per channel statistical analysis.

Click to expand...

You weren't using IRIS correctly, then. IRIS loads the RAW data in colorblind monochrome format with the "Load a RAW file" menu item. It only interpolates full RGB if you "Load" the RAW file, based on your camera settings.

John Sheehy said:

Unless you have a long exposure, or your camera is defective, the blackpoint is 128 for current 1D*, 5D, 20D, and 30D cameras, and 256 for the rebels. The blackpoint may float almost 1 ADU higher for high ISOs, from the tiny bit of dark current present even in normal shutter speeds.

Click to expand...

Oh, I forgot, the blackpoint is 1024 to 1025 ADU for the 1Dmk3. If you load it into IRIS, you can see for yourself as the black strip just to the left of the image is unexposed. Careful, though; there's a vertical grey strip just to the left that is hard to see. You have to lower the threshold max to see it and avoid selecting it to do a "stat" with the right mouse button.

Now that I think of it, your complaint about the blackpoint came back to my mind. Did you not like the fact that the black level was not zero? Clipping to zero is not optimum. Subtracting the black point is useful, and IRIS maintains the negative numbers, which are very useful. If you are going to do any kind of downsampling, binning, white balancing, or "demosaicing" (I don't think any of IRIS' methods are true demosaicing; they are just interpolations), it is best to do them with the negative numbers unclipped (you can actually do all of them except WB before subtracting the blackpoint).

If I do these tests again or test other cameras (and I might since I already have the step wedge and the imatest software) I'll make sure to get the exposure set correctly.

I am impressed by the collective knowledge here at OPF. When you guys start building custom cameras let me know! Actually for now I would love to just figure out how to build a lightroom preset for my DMR that looks good. None of the calibration scripts that I have tried are giving me results that look good or are accurate.

John Sheehy said:

Unless you have a long exposure, or your camera is defective, the blackpoint is 128 for current 1D*, 5D, 20D, and 30D cameras, and 256 for the rebels. The blackpoint may float almost 1 ADU higher for high ISOs, from the tiny bit of dark current present even in normal shutter speeds.

Click to expand...

Let me tell you frankly that I am very sceptical about that 128 blackpoint level. First reason is that it is mathmatically/physically unlikely to have no signal in the Least Significant Bits (LSBs). Second reason is that ImagesPlus gives 16-bit values of 0, 16, 32, 48, ... for the blackframe's Green channel, which obviously is 0, 1, 2, 3, ... in 12-bit space. The other channels have values of Red 0, 36, 73, 110, ... and Blue 0, 20, 40, 60, ... which are all linear slopes before color balancing. The third reason is that it just doesn't make sense to not use the LSBs.

You also need to do a completely blown-out exposure at the same ISO to determine what the RAW clipping point is.

Click to expand...

That does make sense, as does the limiting of the highest saturation Digital Number by Canon, to reduce blooming/fringing artifacts in postprocessing.

Bart