450D noise performance?

anyone looked at it yet?

since IRIS now supports it i gave it a try ... loading a blackframe and doing that 'stats' command.
I got the following results:

Sigma Values:
------- 450D 300D 20D
iso 100 11.3 01.9 03.1
iso 200 11.7 02.9 02.1
iso 400 13.1 05.0 02.5
iso 800 16.9 09.6 03.3
iso1600 25.5 19.1 05.0
iso3200 ---- 19.2 10.0

i found another programm called 'rawanalyse' which gives different values ...
If someone is interested in those raw-files then i'll upload them ...

Torsten Kötting said:

Sigma Values:
------- 450D 300D 20D
iso 100 11.3 01.9 03.1
iso 200 11.7 02.9 02.1
iso 400 13.1 05.0 02.5
iso 800 16.9 09.6 03.3
iso1600 25.5 19.1 05.0
iso3200 ---- 19.2 10.0

Click to expand...

Have you tried black frames from any of the other 14-bit Canons? Would anyone be so kind as to provide them to Torsten?

Those numbers appear disproportionate from the pixel pitch and since the sensors have 4 times the potential dynamic range, dividing the numbers by 4 looks more in line (at lower ISOs) with the decrease in pixel pitch w/ increased resolution.

just curious,

Sean

For the people not in the know, can someone please explain what are we looking at here?
Thanks.

Torsten Kötting said:

anyone looked at it yet?

since IRIS now supports it i gave it a try ... loading a blackframe and doing that 'stats' command.
I got the following results:

Sigma Values:
------- 450D 300D 20D
iso 100 11.3 01.9 03.1
iso 200 11.7 02.9 02.1
iso 400 13.1 05.0 02.5
iso 800 16.9 09.6 03.3
iso1600 25.5 19.1 05.0
iso3200 ---- 19.2 10.0

Click to expand...

Those low ISO sigmas for the 450D seem pretty high. Did you look at the blackframe to make sure that it didn't include parts of the RAW file which weren't image data? Iris usually does that when it first decodes a new format, but hasn't been modified to address the camera fully. Try a rectangle in the middle of the image, and use the right-mouse=clock "statistics" and see if you get the same thing.

John Sheehy said:

Those low ISO sigmas for the 450D seem pretty high. Did you look at the blackframe to make sure that it didn't include parts of the RAW file which weren't image data? Iris usually does that when it first decodes a new format, but hasn't been modified to address the camera fully. Try a rectangle in the middle of the image, and use the right-mouse=clock "statistics" and see if you get the same thing.

Click to expand...

those really are blackframes. Selecting a part of it in iris and hitting statistics gives the same result. But isnt it strange that the iso200 of the 20D is lower than its iso100? i dont know ...
If anyone cares the raws are here: http://www.pipetrap.de/stuff/blackframes/

edit: i wonder how a few bright hotpixels would mess up such a result ... is there a way to tell Iris to give the sigma-value for each color seperately? from long-exposure tests against my trusty 20D i know that the 450Ds weak point there is the red channel. I know 2 other people who have that camera. I'll try to get a few blackframes from them. Maybe i got a faulty camera .... i wouldnt be surprised ... its Canon.

Cem_Usakligil said:

For the people not in the know, can someone please explain what are we looking at here?
Thanks.

Click to expand...

Its a way to analyse the noise-performance of a digicam. You shoot a frame which has a uniform color, black is done the easiest way. Then you look for every single pixel how much it differs from that color (the mean from the hole frame). That difference is caused by noise. The sigma-value contains the (weighted?) mean of those variations from the hole frame, it represents the noise-value. The higher that value, the stronger the noise.
To comepare the values from the 450D (14Bit) with the other cameras (12Bit) you have to multiply the values with 4 to make up for the bit-depth difference.
or better wait for John to explain it the proper way )

Hi Torsten,

Thank you very much for putting things into a perspective for us. Your explanation is quite clear to me. But now that I understand it, I am also worried about it. How is it possible that the sigma values for 450D at ISO to be that high compared to older technology? I realize that the sensor has a higher density but still, the difference is much higher than expected. Or not??

Cheers,

Cem_Usakligil said:

Hi Torsten,

Thank you very much for putting things into a perspective for us. Your explanation is quite clear to me. But now that I understand it, I am also worried about it. How is it possible that the sigma values for 450D at ISO to be that high compared to older technology? I realize that the sensor has a higher density but still, the difference is much higher than expected. Or not??

Cheers,

Click to expand...

yes im a bit disapointed as well. Maybe John could post the results of his 400D. I think there are also sample-variations. Maybe i got a bad one ... the price for the early-adopter you know

heres an example of the rawanalyser-result:

i just repeated the test with the 450D ... same settings. Im getting lower values now ... god knows why:
iso100: 9.7
iso200: 10
iso400: 11.2
iso800: 14.5
iso1600: 21.6

only thing that changed was that i now had no lens attached, only the body-cap. The first test was done with the 50/1.8 lens and lenscap. ... solar flares?
Only thing that comes to mind is power-comsumption? i'll test again with a big IS-zoom later today.
btw. high-iso nr doesnt affect the test-results.
And for fun i put my old 300D into the fridge for 2 hours and redid the test as well ... it didnt affect the results.

Edit: ok found it. For some reason the noise-performance depends on the aperture. Below f3.2 the noise will raise for whatever reason:
all test done with iso1600:
f1.8: sigma 25.4
f2.0: sigma 24.1
f2.2: sigma 22.9
f2.5: sigma 22.5
f2.8: sigma 22.0
f3.2: sigma 21.6
>3.2 sigma 21.6

Torsten Kötting said:

Im getting lower values now ... god knows why:

Click to expand...

I'm not God, but the most likely candidate for such variations is light entering through ...
the viewfinder! I know from experience how easily it is overlooked (unless one takes one's dark frames in the dark). Shooting with the bodycap on, instead of a lens with lens cap, also removes another potential variable.

You'll also get more accurate results if you use 2 dark frames for each ISO, and subtract them from each other (and add an offset to get the result far enough above zero to avoid all clipping at the minimum). IRIS allows to do that. The resulting 'Sigma' should be divided by Sqrt(2) to adjust for the subtraction effect. Doing so, you'll reduce the effects of systematic pattern noise and other non-uniformities, leaving only random noise.

It is always helpful to specify the bit depth (e.g. 12 or 14 bits) of the Raw data, since it allows to adjust before comparison with other sources of info.

Bart

Bart_van_der_Wolf said:

I'm not God, but the most likely candidate for such variations is light entering through ...
the viewfinder! I know from experience how easily it is overlooked (unless one takes one's
dark frames in the dark). Shooting with the bodycap on, instead of a lens with lens cap, also removes another potential variable.

You'll also get more accurate results if you use 2 dark frames for each ISO, and subtract them from each other (and add an offset to get the result far enough above zero to avoid all clipping at the minimum). IRIS allows to do that. The resulting 'Sigma' should be divided by Sqrt(2) to adjust for the subtraction effect. Doing so, you'll reduce the effects of systematic pattern noise and other non-uniformities, leaving only random noise.

It is always helpful to specify the bit depth (e.g. 12 or 14 bits) of the Raw data, since it allows to adjust before comparison with other sources of info.

Bart

Click to expand...

thanks for the hints ... i covered the vf with my thumb when doing these frames. Im pretty sure the aperture-dependend effects arent creeping light or something like that. Maybe you can try that test too. Would be interesting to know if the 1ds does the same ...

Torsten Kötting said:

Edit: ok found it. For some reason the noise-performance depends on the aperture. Below f3.2 the noise will raise for whatever reason:
all test done with iso1600:
f1.8: sigma 25.4
f2.0: sigma 24.1
f2.2: sigma 22.9
f2.5: sigma 22.5
f2.8: sigma 22.0
f3.2: sigma 21.6
>3.2 sigma 21.6

Click to expand...

Other people have reported this in the past. I've seen it on my 400D, too. The RAW data was pushed 1/3 stop in the camera for one frame, I believe wide open. I noticed gaps in the RAW histogram, and realized that the data was stretched. Sure enough, I just looked at your ISO 100 from the 450D, and its histogram was stretched. This means that the gain is actually a bit higher than what you'd expect. Perhaps canon knows something about their lenses having relatively higher t-stops at lower f-stops, and compensates in the firmware. That's a bad idea, IMO, because you lose DR when you do that. They should really just adjust the whitepoint value in the RAW file's metadata, and make the converters take care of it. The gaps themselves are not an issue as far as IQ is concerned; they didn't make any visible difference under normal circumstances in 12 bits, and they certainly won't in 14.

Torsten Kötting said:

i just repeated the test with the 450D ... same settings. Im getting lower values now ... god knows why:
iso100: 9.7
iso200: 10
iso400: 11.2
iso800: 14.5
iso1600: 21.6

Click to expand...

That's still a bit high, though. Even with the extra pixels, that gives the lowest DR of any Canon DSLR from the 10D on, at ISO 100. Looks a little better perhaps than the 40D at ISO 1600, though. This is in terms of DR; in terms of absolute signal-to-readnoise, I can not comment, because the exact improvement in quantum efficiency in the 450D is unknown.

This is an interesting pattern; something similar happened with the Pentax K10D -> K20D upgrade. The K10D had the lowest read noise of any DSLR at base ISO, but it's highest ISO was pretty much the same thing as base ISO pushed. The K20D was interesting me because it was going to be 14MP, and I had hoped that the same low-ISO read technology might be there to give the K20D tremendous DR at base ISo, but it turned out to have one of the worst read noises of any DSLR at base ISO.

The pattern that seems to be emerging is that it is technically difficult to design a single camera to have good high-ISO noise and good low-ISO noise with higher pixel densities. The reason may be as simple as unoptimized, fast readout that punishes the low ISOs more.

If this is the trend of the future, I really hope that manufacturers will start making different bodies for different ends of the ISO spectrum. Implementing compromised high-ISO in a low-ISO body is easy, and all P&S cameras and most existing DSLRs already work that way. Implementing low ISOs in high-ISO bodies is more problematic, because you need neutral density effects for long shutter speeds.

Rest assured, however, that the higher low-ISO noise is not directly due to pixel density per se; there are P&S cameras with 2 micron pixels that have similar read noise. IMO, some kind of compromise issue is involved.

If Canon (or any manufacturer) could make a 100MP APS sensor filled with 2 micron pixels, with uncompromised read speed, while maintaining base ISO read noise at less than 3 12-bit ADUs, the camera could perform as well a current DSLRs at high ISOs, and be much, much better at low ISOs than any current DSLR. That's a long way from this year's technology, though, and with the coarser pixel pitches used in DSLRs, they still need to have designs optimized for high ISOs, for good low-light performance.

Torsten Kötting said:

those really are blackframes. Selecting a part of it in iris and hitting statistics gives the same result. But isnt it strange that the iso200 of the 20D is lower than its iso100? i dont know ...
If anyone cares the raws are here: http://www.pipetrap.de/stuff/blackframes/

edit: i wonder how a few bright hotpixels would mess up such a result ...

Click to expand...

You can clip everything above the top of the histogram, where it tapers of, and redo the stats. I tried it in your 450D ISO 100, and it made no difference.

is there a way to tell Iris to give the sigma-value for each color seperately?

Click to expand...

There is; you can separate the image into 4 channels with the command "split_cfa g2 b r g1", and then "load r", etc, to look at individual channels. Canon readout and digitization, to date, has been totally colorblind, though. There is no treatment of color channels as such until conversion happens in software. However, in many cameras there are problems with uneven gain in odd and even horizontal lines, so when *do* have signal, g2 and b will have different gain than r and g1.

John Sheehy said:

You can clip everything above the top of the histogram, where it tapers of, and redo the stats. I tried it in your 450D ISO 100, and it made no difference.



There is; you can separate the image into 4 channels with the command "split_cfa g2 b r g1", and then "load r", etc, to look at individual channels. Canon readout and digitization, to date, has been totally colorblind, though. There is no treatment of color channels as such until conversion happens in software. However, in many cameras there are problems with uneven gain in odd and even horizontal lines, so when *do* have signal, g2 and b will have different gain than r and g1.

Click to expand...

thanks John, i'll try that.

Torsten Kötting said:

Edit: ok found it. For some reason the noise-performance depends on the aperture. Below f3.2 the noise will raise for whatever reason:
all test done with iso1600:
f1.8: sigma 25.4
f2.0: sigma 24.1
f2.2: sigma 22.9
f2.5: sigma 22.5
f2.8: sigma 22.0
f3.2: sigma 21.6
>3.2 sigma 21.6

Click to expand...

I've done some experimentation on my Rebel XSi (450D), taking multiple exposures to "fill up" the histogram, followed by programmatic analysis of the RAW data, and have found the following:

• The aperture-dependent scaling indeed creates gaps in the raw histogram.
• The only clean set of gaps is in the Red channel. I need to do more data manipulation to figure out why the other channels are different.
• The gaps are not absolute. Occasionally a pixel will have an "illegal" value. I'm not sure why. The frequency of "gap fillers" is very sparse, and is almost random, but has a slight vertical banding to it (like read noise). This seems to indicate that the camera does some things after scaling but before saving to RAW.
• Using a map of these gaps, I can reverse-engineer the scaling algorithm that the camera must be using.

The widest lens I have is F/2.8, so I can't do a full chart. Here's the formula I found for F/2.8:
scaled = 948+((original-949)*16708-46)/16384
This is a 101.977% scaling, consistent with Torsten Kötting's data for F/2.8 and F/3.2+, suggesting that the scaling is lens-independent — but more data is needed to confirm this!

The saturation points of my 450D are consistent with this scaling formula:
F/2.8: 14873 ADU
F/3.5: 14604 ADU
floor(948+((14604-949)*16708-46)/16384) = 14873

David Ellsworth said:

I've done some experimentation on my Rebel XSi (450D), taking multiple exposures to "fill up" the histogram, followed by programmatic analysis of the RAW data, and have found the following:

• The aperture-dependent scaling indeed creates gaps in the raw histogram.
• The only clean set of gaps is in the Red channel. I need to do more data manipulation to figure out why the other channels are different.
• The gaps are not absolute. Occasionally a pixel will have an "illegal" value. I'm not sure why. The frequency of "gap fillers" is very sparse, and is almost random, but has a slight vertical banding to it (like read noise). This seems to indicate that the camera does some things after scaling but before saving to RAW.
• Using a map of these gaps, I can reverse-engineer the scaling algorithm that the camera must be using.

The widest lens I have is F/2.8, so I can't do a full chart. Here's the formula I found for F/2.8:
scaled = 948+((original-949)*16708-46)/16384
This is a 101.977% scaling, consistent with Torsten Kötting's data for F/2.8 and F/3.2+, suggesting that the scaling is lens-independent — but more data is needed to confirm this!

The saturation points of my 450D are consistent with this scaling formula:
F/2.8: 14873 ADU
F/3.5: 14604 ADU
floor(948+((14604-949)*16708-46)/16384) = 14873

Click to expand...

Hi David, Torsten,

Your observations are correct.

This phenomenon - the gaps (combing) in the raw data at f/2.8 and below - has been known for a while. All evidence suggests that it is a correction for the micro-lens efficiency at wide apertures. You can read all about it here:

http://www.openphotographyforums.com/forums/showthread.php?t=5717
and here:
http://forums.dpreview.com/forums/read.asp?forum=1019&message=27936802