PDA

View Full Version : Combing artifacts in 30D RAW histograms - what casues them? (Chuck Westfall?)


Peter Ruevski
July 27th, 2006, 07:18 AM
Hi all,

It has been two and a half months since I bought a 30D (after 14 years with two film SLRs) - it is a wonderful camera!

Being a programmer however I just had to look into the RAW data that the camera generates. One of the reasons I did it is because I had read that the intermediate ISO sensitivities (e.g. ISO 125) on the 30D are achieved by scaling the already digitized signal in the firmware. It indeed turned out to be the case. As a result the RAW histograms of photos taken at intermediate ISOs have quite heavy (dense) combing artifacts – which is not at all surprising.

However I also discovered that the RAW histograms of some photos taken at full stop ISO sensitivities are also combed! Here is an example of a good histogram:

ISO 100, EC 0, 1/80 f/22
http://www.pages.drexel.edu/~par24/rawhistogram/IMG_1753.png

And here is and example of a combed one:

ISO 100, EC +1/3, 1/500 f2
http://www.pages.drexel.edu/~par24/rawhistogram/IMG_0843.png

What the above shows is that even at full ISOs the firmware of the 30D sometimes manipulates the RAW data coming from the A/D converters before writing it to the CR2 file – so the “RAW file” is not really RAW in these cases.

How often does it happen? Well I wrote a program (and a batch file) that can go through any number of CR2 files and check whether their histograms are combed. Using that found that from the 1304 CR2 files that I have – all of them shot at full ISOs - 216 have combed histograms. The period of the combing pattern varies from one photo to the other. I have put together a web page that has many more examples, the software used and some examples from other Canon dSLRs.
http://www.pages.drexel.edu/~par24/rawhistogram/rawhistogram.html

Even though I tried – I was unable to find any correlation between the shooting conditions and the combing appearing in the histograms. I find all of the above very interesting and would really like to hear any ideas or explanations of why it is happening. You can also try it on your own RAW files using the software I wrote.

Best regards,
Peter Ruevski

Chuck Westfall(Canon USA)
July 27th, 2006, 10:03 AM
Hi, Peter:

I appreciate your interest, but we do not comment on our image processing algorithms. Our cameras are basically offered "as is," and we do our best to make sure that image quality is as high as it can be. Canon's official statement on EOS 30D image quality is as follows:

"The image quality of the EOS 30D and all other EOS Digital SLRs conforms with Canon's internal quality standard at all available ISO speed settings. We have no further comments to offer on this issue."

Best Regards,

Chuck Westfall
Director/Media & Customer Relationship
Camera Marketing Group/Canon U.S.A., Inc.

Asher Kelman
July 27th, 2006, 10:21 AM
Hi, Peter:

I appreciate your interest, but we do not comment on our image processing algorithms. Our cameras are basically offered "as is," and we do our best to make sure that image quality is as high as it can be. Canon's official statement on EOS 30D image quality is as follows:

"The image quality of the EOS 30D and all other EOS Digital SLRs conforms with Canon's internal quality standard at all available ISO speed settings. We have no further comments to offer on this issue."

Best Regards,

Chuck Westfall
Director/Media & Customer Relationship
Camera Marketing Group/Canon U.S.A., Inc.
Chuck,

Again, we appreciate your following threads here for us!

Could you state whether or not the 5D or 1D series share identical ways of dealing with luminance? IOW, can we avoid such 30D effects by buying a more professional camera or do all the cameras use the same basic translation mathematics?
Asher

Peter Ruevski
July 27th, 2006, 11:14 AM
Hi Chuck,

Thank you very much for your quick reply and for following user forums.
As for the combing artifacts in round ISO speed RAW histograms - I do not think that they visibly affect image quality. In some almost identical photos where one had the combing and the other did not - I could not see a difference in the image quality. In fact I am very satisfied with the image quality of my 30D! What is more, most post-processing operations do "much worse" things to the data.

So this is not about image quality at all - and I was very careful not to imply such a thing - it is more of an "academic" interest for me - an unsolved mystery in a way. Canon's "no comment" policy is perfectly understandable, I just though no harm can come from asking :-)

As for the intermediate ISOs - I do not mind them at all - and they are not a "mystery". What is more they would certainly be useful for people shooting JPEG - and when shooting RAW one can achieve the effect "consciously" in RAW conversion and just avoid them.

Thank you again for your reply.

Best regards,
Peter Ruevski

P.S. Everyone else - the mystery remains :-)

Chuck Westfall(Canon USA)
July 27th, 2006, 12:46 PM
>>Could you state whether or not the 5D or 1D series share identical ways of dealing with luminance? IOW, can we avoid such 30D effects by buying a more professional camera or do all the cameras use the same basic translation mathematics?<<

Hi, Asher:

Those questions are precisely the ones that would lead to 'further comments' on image quality that Canon Inc. is not willing to discuss. Sorry!

Best Regards,

Chuck Westfall
Director/Media & Customer Relationship
Camera Marketing Group/Canon U.S.A., Inc.

Stan Jirman
July 27th, 2006, 02:03 PM
Maybe Peter can run his batch program on a series of CR2 files from a 5D and some 1D-series cameras? Then we won't have an official Canon statement, but empirical evidence.

Harvey Moore
July 27th, 2006, 02:08 PM
It appears to me that you are analysing dpraw interpretation of the files, not the actual raw data.

Asher Kelman
July 27th, 2006, 02:09 PM
Maybe Peter can run his batch program on a series of CR2 files from a 5D and some 1D-series cameras? Then we won't have an official Canon statement, but empirical evidence.

Stan, I was thinking the same thing. I do check my histiograms and I don't recall seeing combs on my 5D images. I'll look out for them in my current processing.

Peter, how many files would you need? Which ones?

Asher

Nill Toulme
July 27th, 2006, 02:57 PM
My grandfather always used to say "Never trust a man with comb tracks." Is that what y'all are talking about?

Nill
~~
www.toulme.net

Peter Ruevski
July 27th, 2006, 03:05 PM
@Harvey Moore

Indeed I am analyzing the output of dcraw, however I am using it in the mode in which it outputs unmodified RAW data (not even Bayer demosaicing is done) to a 16bit PGM file. I posted a similar thread on dpreview (http://forums.dpreview.com/forums/read.asp?forum=1019&message=19009766) and a similar question came up. I have traced through (debugged) the dcraw code path that outputs the RAW data and I believe that it is indeed untouched 12bit raw data as it came from the ADCs stored in 16bit values inside the PGM.

Bottom line - I am 99.9% certain that the combing is not introduced by dcraw.

@Asher Kelman

I would love to analyze some 5D files. Ideally it would be the same scene shot under different ISO settings - including 1/3 steps - and preferably one that you would not mind me posing the pictures of on the page I mentioned in the original post.

Keep in mind however that the combing may be a rare occurrence - I have a friend with a 350D, he ran the search on about 500 of his CR2 files and only two showed combing.

If you are using a PC (Windows) you can PM me and I can send you the required files and instructions so you can run the search on a big collection of files.

As for seeing the combing in the histograms - I am talking about histograms of the RAW data - even processing the RAW file to a 16 bit linear TIFF would already hide the effect - the bayer demosaicing is in effect an interpolation. The only other program that I am aware of (apart from mine) that can show the real RAW histogram is Iris http://www.astrosurf.com/buil/us/iris/iris.htm - and it shows the combing too. See this (http://forums.dpreview.com/forums/read.asp?forum=1019&message=19131963) for an explanation how to use it - it has a GUI so it is easier to use (and immensely more powerful) than my program. The only advantage of mine is that it can be used to do an automated search through many files.

Asher Kelman
July 27th, 2006, 03:20 PM
Peter,

Impressive work. I'm embedded in Mac computers! It would be great if XP users could volunteer to test.

Asher

Peter Ruevski
July 27th, 2006, 03:29 PM
My grandfather always used to say "Never trust a man with comb tracks." Is that what y'all are talking about?


You mean "Italian Mafia style" hairdo - lots of gel and combed backwards? :-)

No, we are talking about those equally spaced vertical lines (in fact they are single values that are zero) in the second histogram above. It is colloquially called combing because it looks like a comb. It happens for example when you “stretch the histogram” of an image (essentially multiplying each pixel value by a number). Imagine a simple monochrome image, which only has pixels with five shades of gray 0, 1, 2, 3 and 4. If you multiply every pixel by 2 now your image will be "twice as bright" (in fact it will have twice the dynamic range). But it will only have pixels with values 0, 2, 4, 6 and 8 – there will be no pixels whatsoever with values 1, 3, 5 and 7. If you build the histogram every odd value will be 0.

But who knows - maybe the Mafia has something to do with this too ;-)

Guy Tal
July 27th, 2006, 05:41 PM
If you got a pre-emptive "here's our official position" from Canon, you must be on to something here :)

I'm curious if you analyzed files captured at ISO 50 ("Low" setting once you enable ISO expansion). I was always curious whether lower ISO will actually provide some benefit other than extending exposure time.

Guy

Doug Kerr
July 27th, 2006, 07:30 PM
Hi, Peter,

Very interesting. Thanks for the report and the clear presentation of your findings.

It will be fascinating to try and develop some conjecture to explain thiese results!

Asher Kelman
July 27th, 2006, 08:01 PM
Hi Doug,

I knew you'd arrive! Here's the challenge. What are the limits if any to dcRAW combed files?

A print from a "combed" versus a normal file might print the same to an 8x10, but how does the image withstand various filters, B&W conversions, upressing or sharpening?

I have a feeling that the file might be less robust.

Asher

BobSmith
July 28th, 2006, 04:57 AM
I have a feeling that the file might be less robust.

Possibly... but only if the added data in the non combed version is truly meaningful image data and not just noise. This layman would think that an accurate analysis of that might be very difficult to do without more knowledge of Canon's proprietary inner workings.

Peter Ruevski
July 28th, 2006, 07:13 AM
Hi all,

I will try to answer in a single post so that the thread does not get unnecessarily littered with my posts.

First let me say that I would love to look at any sample RAW files that anyone is willing to send me. For those that do not have any web space to host big files you can use http://www.yousendit.com/ - it is a service for sending big files. Just put your own email as the recipient, upload the file and when the upload is done it will give you a unique link for downloading the file. Email me that link and let me know if you just want me to analyze it and let the forum know how it looks or I can also post it as an example.

If you got a pre-emptive "here's our official position" from Canon, you must be on to something here :)

Well I am "on to something" but it may not be serious at all – they may simply be worried that people get the wrong message – e.g. that the image quality is compromised – which I actually do not think it is – at least not visibly.

I'm curious if you analyzed files captured at ISO 50 ("Low" setting once you enable ISO expansion). I was always curious whether lower ISO will actually provide some benefit other than extending exposure time.

The 30D does not provide ISO 50 when the ISO expansion is enabled – only ISO 3200 – and that is just ISO 1600 pushed a stop – so not interesting. If you send me an ISO 50 file I would certainly look at it.

Hi, Peter,

Very interesting. Thanks for the report and the clear presentation of your findings.

It will be fascinating to try and develop some conjecture to explain thiese results!

Absolutely! I spent quite a bit of time trying to, but so far I do not have any even remotely defensible conjecture to present.

A print from a "combed" versus a normal file might print the same to an 8x10, but how does the image withstand various filters, B&W conversions, upressing or sharpening?

I have a feeling that the file might be less robust.

Possibly... but only if the added data in the non combed version is truly meaningful image data and not just noise. This layman would think that an accurate analysis of that might be very difficult to do without more knowledge of Canon's proprietary inner workings.

On this I tend to agree with Bob – combing is a "fact of life" in post processing – the first time you do for example Levels, Curves, Brightness/Contrast in Photoshop – you rare doing "much worse" things to the data in the image. So the combing in the raw will hardly matter for image quality – in fact it is gone as soon as the interpolation to an RGB image from the Bayer (CFA) image is done during RAW processing.
The above does not make it any less interesting though - the the combing certainly appears because the firmware did something to the RAW data. Unless this "something" is just an obscure firmware bug then they (Canon engineers) must have felt it is necessary - and they have a proven track record of not touching the RAW data for nothing - so it may be compensating for some deficiency/effect in the image pipeline.

And finally (as a consolation and one more reason to like Canon :-) all Nikon RAW files are like that - much worse in fact. Look at the histograms halfway down this page:
http://www.astrosurf.com/buil/d70v10d/eval.htm
in the "6. CODAGE NUMERIQUE / NUMERICAL CODING" section. And more information here:
http://www.majid.info/mylos/weblog/2004/05/02-1.html

Best regards,
Peter

Richard McNeil
July 28th, 2006, 02:32 PM
I have used the IRIS program and looked at ~200 cr2 files with iso's from 100 - 1600. No combing artifacts at all. Will run a better (more controlled) test this weekend.

Richard McNeil
July 28th, 2006, 02:33 PM
I have used the IRIS program and looked at ~200 cr2 files with iso's from 100 - 1600. No combing artifacts at all. Will run a better (more controlled) test this weekend.

Should have added using the EOS 5D.

Daniel_Hyams
July 28th, 2006, 02:53 PM
Fascinating stuff! I'll certainly download the source, compile, and see what my photo repository looks like.

Richard McNeil
July 29th, 2006, 11:43 AM
I took my 5D with a 100mm F2.8 Macro lens to the garden this morning and photographed a flower at all ISO settings (100 through 1600). The camera (firmware 1.1.0) was set to aperture priority, F16, MLU and I used the remote switch. I viewed all histograms using the IRIS program and saw no combing artifacts at all!

I will try ruevs program and procedure this afternoon and let you know what happens.

Daniel_Hyams
July 30th, 2006, 12:17 PM
If you want to process an entire directory of raws, here is a quick and dirty batch file to do this under windows. Just name it process.dat (or anything, really), and make sure that both dcraw.exe and rawhistogram.exe are in your search path somewhere (or in the same directory)

=====cut here

for %%i in (*.CR2) do dcraw -D -4 -t 0 -o 0 -v %%i
for %%i in (*.pgm) do rawhistogram %%i

=====cut here

Not pretty, but it will get the job done ;) Unix provides much better scripting facilities... O.O

Peter Ruevski
July 31st, 2006, 10:44 AM
If you want to process an entire directory of raws, here is a quick and dirty batch file to do this under windows. Just name it process.dat (or anything, really), and make sure that both dcraw.exe and rawhistogram.exe are in your search path somewhere (or in the same directory)

=====cut here

for %%i in (*.CR2) do dcraw -D -4 -t 0 -o 0 -v %%i
for %%i in (*.pgm) do rawhistogram %%i

=====cut here

Not pretty, but it will get the job done ;) Unix provides much better scripting facilities... O.O

Hi Daniel and everyone else,

This is almost exacly what I did. And the DOS batch language is indeed quite limited. Here is my batch file:

=====cut here

@echo off

for /R .\ %%f in (*.cr2) do (
dcraw.exe -D -4 -t 0 -o 0 -v %%f )

for /R .\ %%f in (*.pgm) do (
rawhistogram.exe %%f & (if ERRORLEVEL 1 echo Combed>%%f.flg) & del /q %%f
)

echo List of combed RAW files:> ComedRAWList.txt
echo ------------------------->> ComedRAWList.txt
dir /s /b *.flg >> ComedRAWList.txt

=====cut here

You can use it to easily check many raw files for combing at once. Let us presume that your raw files are stored in a directory "C:\rawfiles" and in many sub-directories.

1. Place "findcombedraws.bat", "dcraw.exe" and "rawhistogram.exe" in the C:\rawfiles directory.
2. Start a command prompt – under Windows XP choose "Start | Run", type "cmd" an press OK.
3. At the command prompt type:
cd c:\rawfiles
and press enter.
4. Type:
findcomedraws.bat
and press enter.
5. The batch file will go through the current directory and all its sub-directories, find all CR2 files, convert them to PGM and then check whether their histograms are combed. This will take a while if you have many RAW files.
6. A text file "ComedRAWList.txt" that lists all of the combed RAW files found will be created in C:\rawfiles.
7. In addition for each *.cr2 file there will be one *.CSV file created that contains the RAW histogram data - you can plot this file in Excel. For all combed *.cr2 files there will be a small text file called *.pgm.flg that simply marks that the corresponding RAW file is combed. The *.PGM files will be deleted to save space.

Since the batch file may be useful and make it easier for peole to search their raw file collections I made a package that contains everything needed and placed it here:
http://www.pages.drexel.edu/~par24/rawhistogram/findcombedraws.zip
I also updated the page to offer a download of this file.

Hopefully this will stimulate more people to search (completely automatically) through their raw files.

Best regards,
Peter

Peter Ruevski
July 31st, 2006, 10:51 AM
I took my 5D with a 100mm F2.8 Macro lens to the garden this morning and photographed a flower at all ISO settings (100 through 1600). The camera (firmware 1.1.0) was set to aperture priority, F16, MLU and I used the remote switch. I viewed all histograms using the IRIS program and saw no combing artifacts at all!

I will try ruevs program and procedure this afternoon and let you know what happens.

Hi Rechard,

So I presume when you said "all ISO settings" you included intermedate 1/3 stop ones like ISO 125? If so and you did not see any combing then the 5D would certainly seem to have "proper" i.e. alanog amplifier driven intermediate ISO settings.

...unlike the 30D. Here is what an ISO 125 histogram looks like from the 30D:
http://www.pages.drexel.edu/~par24/rawhistogram/IMG_1754.png
The histogram looks "solid" because of the heavy combing. Here is a zoomed in section:
http://www.pages.drexel.edu/~par24/rawhistogram/IMG_1754_zoom.png

Best regards,
Peter

Richard McNeil
July 31st, 2006, 11:47 AM
Hi Rechard,

So I presume when you said "all ISO settings" you included intermedate 1/3 stop ones like ISO 125? If so and you did not see any combing then the 5D would certainly seem to have "proper" i.e. alanog amplifier driven intermediate ISO settings.

...unlike the 30D. Here is what an ISO 125 histogram looks like from the 30D:
http://www.pages.drexel.edu/~par24/rawhistogram/IMG_1754.png
The histogram looks "solid" because of the heavy combing. Here is a zoomed in section:
http://www.pages.drexel.edu/~par24/rawhistogram/IMG_1754_zoom.png

Best regards,
Peter

Peter,

Yes i used all ISO settings including iso 125. Let me run your batch file and see what it uncovers.

Richard

Richard McNeil
July 31st, 2006, 12:21 PM
None were flagged as combed!

Peter Ruevski
July 31st, 2006, 12:41 PM
None were flagged as combed!

This is great! How many files did you analyze? It confirms once again that the 5D imaging pipeline is indeed better than the 20D/30D one.

Jason C Doss
July 31st, 2006, 01:48 PM
So, Peter (or anyone else, really), do you think that the combed histograms in the intermediate ISOs on the 30D have a detrimental effect on image quality? Also, what percentage of intermediate ISO 30D files have combed histograms (I am under the impression that it is 100%).

Richard McNeil
July 31st, 2006, 01:53 PM
This is great! How many files did you analyze? It confirms once again that the 5D imaging pipeline is indeed better than the 20D/30D one.
I only did a quick 200 image test - not exactly sure of the ISO distribution. Unfortunately my test files are home and will have to wait till tonight; however, I don't expect to see any combing in those either. Will post results tonight.

Richard

Richard McNeil
July 31st, 2006, 06:43 PM
I have do the test at all iso settings from 50 to 3200 and had no combing artifacts at all! That surprised me because I expected to see them at 50 and 3200 ISO!

Richard

Mike Robbins
August 1st, 2006, 01:45 AM
I just did a scan of a sample of my images (approximately 150 from my old Rebel XT and 1000 from my 5D) using your software, and I found 36 combed RAW files (and only one from the XT). It may be interesting to note that all of the combed files have two things in common: they were all shot at high ISO (800 or higher, with the majority shot at 1600) and they were all shot using my Canon 50 mm f/1.4 lens. Going strictly by memory, I would say that a good number of the shots were also taken using AI Focus mode.

I also did a more controlled test (tripod, MLU, timer) of the entire (non-expanded) ISO range on my 5D, shooting the same subject under uniform lighting. The results of this test showed that none of the RAWs were combed, suggesting that there's something else at work beyond a simple scaling of the RAW data to provide intermediate ISO values.

Peter Ruevski
August 1st, 2006, 09:11 AM
I have do the test at all iso settings from 50 to 3200 and had no combing artifacts at all! That surprised me because I expected to see them at 50 and 3200 ISO!
Hi Richard, The ISO 3200 lacking combing does not surprise me – it is well established that ISO 3200 a.k.a. "H" is simply ISO 1600 pushed (i.e. underexposed) one stop and a flag set in the file telling the raw conversion software to do +1 stop exposure adjustment. As far as the actual raw data is concerned ISO3200 is exactly the same as ISO1600 - that is why people say it is not useful when shooting raw. When shooting JPEG the firmware does the pushing so it is useful.

I do not know how ISO 50 is implemented in the 5D – I do not remember reading anything about it. Could it be just ISO 100 overexposed by one stop? It is easy to find out – just put the camera in manual mode on a tripod in front of a gray card/white wall and shoot two photos – one at ISO 50 and one at ISO 100 – with the same aperture and shutter settings. Then build the raw histograms and compare them. If they look the same then ISO 50 is just ISO 100 overexposed.
I just did a scan of a sample of my images (approximately 150 from my old Rebel XT and 1000 from my 5D) using your software, and I found 36 combed RAW files (and only one from the XT).
So that means you have 35 combed raw files from the 5D?! Would you mind sending me a few examples? And maybe the one from Rebel XT as well? The effect is certainly not common:
- for your 5D - 35 out of a 1000 is 3.5%
- for your 350D - 1 out of 150 is 0.7%
- for my friend’s 350D is was similar to yours
- for my 30D - 216 out of 1304 is 16.6%
It may be interesting to note that all of the combed files have two things in common: they were all shot at high ISO (800 or higher, with the majority shot at 1600) and they were all shot using my Canon 50 mm f/1.4 lens. Going strictly by memory, I would say that a good number of the shots were also taken using AI Focus mode.
This is quite interesting. I have been trying to find a correlation since day one. The 50mm f/1.4 lens is certainly a convincing correlation - but on the other hand if you shoot with it all the time it is no correlation at all. For my 30D photos I have pretty much ruled out ISO speed (excluding intermediate ones), shutter and aperture individually as contributing factors – I have both "good" and "bad" files for all kinds of settings of all three. Of course they may be interacting in a "strange" way. I have not looked at the lens used as closely - I should.
I also did a more controlled test (tripod, MLU, timer) of the entire (non-expanded) ISO range on my 5D, shooting the same subject under uniform lighting. The results of this test showed that none of the RAWs were combed, suggesting that there's something else at work beyond a simple scaling of the RAW data to provide intermediate ISO values.
This is a good test and it confirms Richard’s results. It shows that all the ISO speeds are properly implemented in the 5D. And since you still have combed raw files (as does my 30D at round ISO values) - as you said it shows that something more complex is at work and the combing does not seem to be due to simple scaling.

Which leads me to:
So, Peter (or anyone else, really), do you think that the combed histograms in the intermediate ISOs on the 30D have a detrimental effect on image quality?
At least to me - no visible one, but I do not consider myself to have a sharp/discerning eye :-) On the other hand they are certainly not raw data. What is more when shooting RAW they are not really needed. So I personally do not use them.
Also, what percentage of intermediate ISO 30D files have combed histograms (I am under the impression that it is 100%).
Your impression is correct - 100%. The combing is different though - e.g. ISO125 is “good old down to zero” combing as the examples I posted earlier. ISO160 however is a "half hearted not quite down to zeroes" kind of combing – see this example:
http://www.pages.drexel.edu/~par24/rawhistogram/IMG_1755.png
http://www.pages.drexel.edu/~par24/rawhistogram/IMG_1755_zoom.png
My program does not detect ISO160 type combing - automatically .

Mike Robbins
August 1st, 2006, 09:20 AM
... This is quite interesting. I have been trying to find a correlation since day one. The 50mm f/1.4 lens is certainly a convincing correlation - but on the other hand if you shoot with it all the time it is no correlation at all. ...
Well, I don't have a hard figure as to how many of my shots were taken with each lens, but the majority of them were taken with the 24-105 L. If I had to guess, I'd say that maybe 10-15% of them are with the 50 1.4. Also, it was very rare for me to shoot above ISO 800 on the Rebel XT due to noise, so perhaps that's why there's not so much evidance of the combing.

I have a suspicion that the combing might be to interference from the AF motor / circuit. I have noticed some horizontal banding occasionally when shooting with the 50 1.4 at high ISO in AI Focus mode on my 5D. After I noticed the problem I have not shot with AI Focus very much if at all; the timing of this realization corresponds with the combed files found.

I will see if I can reproduce this behaviour, and run the scan again. Also, I'd be more than happy to send along some of my photos, provided you do not mind looking at shots from a family birthday party. ;)

Richard McNeil
August 1st, 2006, 09:40 AM
I just did a scan of a sample of my images (approximately 150 from my old Rebel XT and 1000 from my 5D) using your software, and I found 36 combed RAW files (and only one from the XT). It may be interesting to note that all of the combed files have two things in common: they were all shot at high ISO (800 or higher, with the majority shot at 1600) and they were all shot using my Canon 50 mm f/1.4 lens. Going strictly by memory, I would say that a good number of the shots were also taken using AI Focus mode.

I also did a more controlled test (tripod, MLU, timer) of the entire (non-expanded) ISO range on my 5D, shooting the same subject under uniform lighting. The results of this test showed that none of the RAWs were combed, suggesting that there's something else at work beyond a simple scaling of the RAW data to provide intermediate ISO values.

I just checked my "old" files and the first picture I took with the 5d used a 50mm f1.8 lens shot at iso 100, f3.2 and in sRGB color space. IT HAS COMBING ARTIFACTS!

I don't think the color space has anything to do with it but I used Adobe color space for all other photos that I tested.

Mike Robbins
August 1st, 2006, 11:00 AM
I performed a quick and somewhat informal test this morning to see if I could reproduce the combing that was found in my earlier scan. I did two series of shots using my 5D, shooting first with my 25-105 mm f/4 and again with my 50 mm f/1.4 lens. Both series of shots were taken at a variety of different ISO values (800-1600) in batches of approximately ten shots per ISO. I shot all of the photos in Av mode (f/4 for the 24-105 and f/2 for the 50) with evaluative metering, all handheld. For the shots taken in AI Focus mode I was sure to move forward or back slightly leading up to the shot to force the focus motors to track; all other shots were taken in one shot mode. The resulting photos are all garbage, but the results may be of some interest (we'll call this first batch Test A):

Test A:

5D, 25-105 @ 50 mm (IS disabled):
ISO 800: 0/10 combed
ISO 1000: 0/10
ISO 1250: 0/10
ISO 1600: 0/10
ISO 1600, AI Focus: 0/11

5D, 50 1.4:
ISO 1600, AI Focus: 0/10
ISO 1600: 3/10
ISO 1250: 2/10
ISO 1000: 1/10
ISO 800: 3/10

I was encouraged to see that I was able to reproduce the problem, though I was surprised to see that there were no combed photos taken with the 50 1.4 in AI Focus mode. I then performed a second set of test shots (Test B), all at ISO 1600 to look more closely at the effect of the focus mode (Av mode again, this time f/1.8; evaluative metering). For the first three batches, I was moving slightly just before I took them to force the micro ring USM to track for focus. The final batch of shots was taken on a tripod, in single shot mode with the timer and MLU. Results are as follows:

Test B:

5D, 50 1.4:
ISO 1600, Single Shot: 11/15
ISO 1600, AI Focus: 12/15
ISO 1600, AI Servo: 8/15
ISO 1600, Single Shot w. Tripod: 14/15!

I am now quite convinced that the combing is a result of the lens / body combination, though I am no longer so sure that it has anything to do with the focus mode. If I get a chance later on, I will check to see if I can produce this combing with my other lenses.

Ray West
August 1st, 2006, 03:28 PM
So, if a raw file is not always a raw file, and iso setting is not as it should be, is this not the same sort of situation as with compact flash capacity not being as on the label. Class action coming, anyone??

Is it similar with Nikon, whoever?


Best wishes,

Ray

Peter Ruevski
August 1st, 2006, 04:18 PM
I performed a quick and somewhat informal test...

Great job Mike you are definitely on to something!

... Test A:....
Test B:...
...
I am now quite convinced that the combing is a result of the lens / body combination, though I am no longer so sure that it has anything to do with the focus mode. If I get a chance later on, I will check to see if I can produce this combing with my other lenses.
So it seems. Time to go back through my files and look for correlation between lens and combing. I downloaded EXIFTool - looks like the perfect tool for the job... now on to writing more batch files to extract and analyze EXIF information of combed files...

For now I can say for sure that I have combed files with the EF 50mm f/1.8 and the EF 100mm f/2 USM.
I also own the EF 24-85mm f/3.5-4.5 USM and the (kit lens; horror! ;-) EF-S 18-55mm f/3.5-5.6 and do not see combed pictures from them - however I only have access to the few photos that are on the web site right now. Will have to check my whole collection later...

Peter Ruevski
August 1st, 2006, 04:30 PM
So, if a raw file is not always a raw file, and iso setting is not as it should be, is this not the same sort of situation as with compact flash capacity not being as on the label. Class action coming, anyone??

Is it similar with Nikon, whoever?

Oh it is much worse with Nikon - most NEF (Nikon RAW) files under the sun are not raw at all - first noise reduction is applied, then the file compression itself is not lossless so practically every NEF is combed.

But class action? Oh, come on, the camera works fine as far as the end result - the picture - is concerned. Which is not to say there isn't a small screw up somewhere - but I am willing to sympathize with the Canon engineers - being an engineer myself - these cameras are a complex piece of hardware/firmware! But we will try to get to the bottom of this anyway - wont we? ;-)

Ray West
August 1st, 2006, 04:56 PM
Hi Peter,

Its not an engineering issue, all engineers are entirely honest, - it is a marketing issue. I do not want to labour the point, I was surprised at the recent cf (and epson ink) actions, but to me this is much the same - something being said that is accepted by most folk, but on examination is a lie. It happens all the time with most things, from a hamburger to a merc, but we somehow seem to accept it.

We will see what happens...

Best wishes,

Ray

Peter Ruevski
August 3rd, 2006, 01:47 PM
Hi all,

So after Mike’s revelation that it might be the lens I just had to go back and check my combed photos. Being a geek - and lazy - I decided that it is too much work to do it by hand so I downloaded ExifTool by Phil Harvey (http://www.sno.phy.queensu.ca/~phil/exiftool/). What can I say… it is the mother of all EXIF utilities; the BFG of meta-data extraction; the Pan Galactic Gargle Blaster of EXIF tools;

the…
One Tool to rule them all, One Tool to find them,
One Tool to bring them all and in the darkness bind them

Did I mention I was impressed? :-) This thing will suck the last bit of metadata out of whatever image file you throw at it!

To make a long story short - after installing Perl; figuring out which of the endless ExifTool options will dump the data (from 775 raw files 215 of wich combed) the way I wanted it; some more batch files and some tinkering in Excel I can say this:

The combing is caused either by the lens, or a combination of the lens and a small aperture. ISO speed, shutter and exposure compensation do not matter it seems. My EF50mm f/1.8 and EF100mm f/2 USM "comb" quite often. My other lenses never do.

I have updated the web page with the full statistics and detailed conclusions as well as some of Mike’s combed 5D photos with the EF50mm f/1.4 USM. Look at the bottom.
http://www.pages.drexel.edu/~par24/rawhistogram/rawhistogram.html

Best regards,
Peter

Ray West
August 3rd, 2006, 02:24 PM
Hi Peter,

Roughly speaking, am I right in thinking that the combing is related to the amount of light falling on the sensor, i.e. more light, tendency to comb? Are the variations due to sensor temperature? Can you get combing with non-canon lenses, say?

Is it worth persuing it further? I think it may take some time to get the reason/cause behind it. I think it may possibly be due to time taken for ad conversions, if the ad works on the capacitor charge-discharge/time principle. i.e. it takes longer to collect the value for a bright image, so we'll do every other one, then, if time left over, go back and fill in the gaps...... Have you tried it with a digic 1 uP based camera?

Wish I had time,

Best wishes,

Ray

Peter Ruevski
August 4th, 2006, 02:01 PM
All very good questions. For all of them the correct answer of course is "I do not know" but the whole point is to build a theory so...

Roughly speaking, am I right in thinking that the combing is related to the amount of light falling on the sensor, i.e. more light, tendency to comb?
I doubt it. I have photos (intentionally done tests) almost completely blown to white (sensor saturated, all pixels at 4095) with a small dark area which show perfectly smooth combing-free histograms.

Are the variations due to sensor temperature?
Possible; however I have a counter example of sorts - IMG_1829 and IMG_1830 on the site. Shot within a minute of each other - one combed, one not. The camera did not move. The only difference - aperture and shutter speed. It is unlikely the sensor temperature could have changed significantly.

Can you get combing with non-canon lenses, say?
I do not know. I only have five photos with a Sigma 170-500 - none of them combed. Now the lens is back to Sigma to be "rechipped" so the aperture works with the 30D. Also a few photos with no lens (a pinhole lens cap) - no combing. But this is a tiny sample.

Is it worth persuing it further? I think it may take some time to get the reason/cause behind it.
Not until there is more data to look at. But that depends on people looking at their collections of raw files. However the issue is an obscure "geeky" one I must confess :-)

I think it may possibly be due to time taken for ad conversions, if the ad works on the capacitor charge-discharge/time principle. i.e. it takes longer to collect the value for a bright image, so we'll do every other one, then, if time left over, go back and fill in the gaps......
Hm-m-m-m - only Canon knows... if the rumors that the ADCs are "one per pixel" are correct (BTW it sounds unlikely to me) then they should be small ones (transistor count wise) - so Sigma/Delta is probable. OTOH if there are "just a few" of them - they must be very fast ones => "proper" parallel conversion ADCs. Either way - why does my EF24-85mm f/3.5-4.5 USM never comb?

Have you tried it with a digic 1 uP based camera?
No, I do not have one.

Regards,
Peter

Daniel_Hyams
August 6th, 2006, 01:18 PM
On a directory of 97 30D files, here is a quick result....

none of the "regular ISO" files were combed. The only ones that were shots that I took at ISO 250, and some were detected by the code as being combed, and some were "lazy combed" as described on the web page.

Lenses used in the shoot were the Tokina 12-24, Tamron 28-75, and Canon 70-200 f/4L.

Alan T. Price
August 30th, 2006, 07:59 PM
It's not surprising to me that there is combing. In fact, it would be more surprising if there was no combing.

The sensor captures light at each pixel only at its native ISO sensitivity and converts the amount of light to an analog voltage. This is then converted to digital in a 12-bit number, allowing for up to 4096 discrete levels.

At higher ISOs the ISO 100 data needs to be scaled up in some way, but there are three ways this can be done:

1. Do a calculation on the 12-bit data and store that new value.

e.g. for a 1-stop increase the values are doubled. This would result in severe combing because only even values can be stored. [any integer multiplied by 2 is even].

If larger increases are applied (e.g. 5 stops) then only every fifth value can be used in the final output.

Clearly there would be so many gaps in the data that this method cannot be the right one.

2. Do a voltage multiplication / amplification in the analog system. This would give no combing because the analog data is not quantised (?) into integers. Simple voltage doubling circuits could be applied for each whole-stop increase in ISO. For that matter the 1/3 and 1/2 stop amplifiers could be used too.

3. do a combination. e.g. Double the analog signal as often as required for whole-stop increases in ISO, but use a digital calculation for the fractions of a stop. With a partial-stop digital multiplication, there would be a gap every few levels as a non-integer result is rounded up or down to fit in the appropriate integer bin.

I'd probably go with #3 for the 30D, to match your observations. Perhaps the bigger cameras have room for more or faster analog circuits to produce less combing, but I haven't seen the observed results and can't make a conclusion.

Quite clearly, this whole experience shows the benefit of having more levels to play with at 12-bit than at 8-bit, as by the time we dumb the data down to 8-bit for printing there is no combing evident.



[Edit:] Here's another possibility: Maybe the whole thing is done digitally but rather than based on a 12-bit initial value, it is based on a 16-bit initial value, as if a 16-bit analogue to digital converter is used. Consider this as being 2^16 quarter-integer levels instead of 2^12 whole-integer values. After scaling, the new 16-bit values are converted to 12-bit values for use in the raw data file. This could be done simply by taking any four adjacent levels at 16-bit depth and puting them into a single level at 12-bit depth, for each pixel.

Third and half stop increases would still lead to combing.

A 16-bit value allows for a 4-stop increase over the conceptual 12-bit value, corresponding to a maximum sensitivity of ISO 1600 if the sensor has a native rating of ISO 100.

Just a thought.

Asher Kelman
August 30th, 2006, 08:12 PM
Thanks Alan for revisiting this issue. Your comments appear valid, however, that is not my field. I'm interested in the responses. I think I'm going to learn some more.

Asher

Peter Ruevski
September 5th, 2006, 09:20 AM
I had to move the page to a new URL:
http://ruevs2.tripod.com/rawhistogram/rawhistogram.html
The old one will stay for some more time but will soon disappear.


Hi Alan,

Your thoughts are quite reasonable and make sense, but (it seems) not all apply to the Canon DSLRs specifically.

It's not surprising to me that there is combing. In fact, it would be more surprising if there was no combing.
With raw files from all Canon DSLRs that I have looked at (300D, 350D, 10D, 20D, 30D, 5D) there is no combing most of the time. The only exceptions that I am aware of are intermediate ISOs on 30D and the phenomenon we were discussing in this thread, which happens at FULL ISOs.

The sensor captures light at each pixel only at its native ISO sensitivity and converts the amount of light to an analog voltage. This is then converted to digital in a 12-bit number, allowing for up to 4096 discrete levels.
Yes

At higher ISOs the ISO 100 data needs to be scaled up in some way, but there are three ways this can be done:
True

1. Do a calculation on the 12-bit data and store that new value.
e.g. for a 1-stop increase the values are doubled. This would result in severe combing because only even values can be stored. [any integer multiplied by 2 is even].

If larger increases are applied (e.g. 5 stops) then only every fifth value can be used in the final output.

Clearly there would be so many gaps in the data that this method cannot be the right one.
This is certainly an option but as you pointed out a very bad one - high ISO becomes meaningless. And it certainly is not the one actually used.
The one exception is 1/3 stop ISO increments on the 30D where the intermediate stops are achieved in this way (with the dark current taken into account) from the closest properly amplified full stop. I will sit down and write a page describing this in detail some day.

2. Do a voltage multiplication / amplification in the analog system. This would give no combing because the analog data is not quantised (?) into integers. Simple voltage doubling circuits could be applied for each whole-stop increase in ISO. For that matter the 1/3 and 1/2 stop amplifiers could be used too.
And this is the method that is apparently used. Including the 1/3 stop increments in the 5D and higher end cameras.

3. do a combination. e.g. Double the analog signal as often as required for whole-stop increases in ISO, but use a digital calculation for the fractions of a stop. With a partial-stop digital multiplication, there would be a gap every few levels as a non-integer result is rounded up or down to fit in the appropriate integer bin.

I'd probably go with #3 for the 30D, to match your observations. Perhaps the bigger cameras have room for more or faster analog circuits to produce less combing, but I haven't seen the observed results and can't make a conclusion.
You are correct.

Quite clearly, this whole experience shows the benefit of having more levels to play with at 12-bit than at 8-bit, as by the time we dumb the data down to 8-bit for printing there is no combing evident.
True

Here's another possibility: Maybe the whole thing is done digitally but rather than based on a 12-bit initial value, it is based on a 16-bit initial value, as if a 16-bit analogue to digital converter is used. Consider this as being 2^16 quarter-integer levels instead of 2^12 whole-integer values. After scaling, the new 16-bit values are converted to 12-bit values for use in the raw data file. This could be done simply by taking any four adjacent levels at 16-bit depth and puting them into a single level at 12-bit depth, for each pixel.

Third and half stop increases would still lead to combing.

A 16-bit value allows for a 4-stop increase over the conceptual 12-bit value, corresponding to a maximum sensitivity of ISO 1600 if the sensor has a native rating of ISO 100.

Just a thought.
Well, it is certainly a possibility but I consider it unlikely. 16bit ADCs are significantly more expensive (both price wise and to design low noise circuits for) than 12 bit ones and amplifiers are quite simple (cheap) by comparison. If Canon had gone to the trouble of putting 16 bit ADCs in they certainly would have advertised the fact loudly and kept the amplifiers to provide higher ISOs instead of wasting the 16bit data with scaling. In fact I think I have seen a direct statement from Cannon that they use 12bit ADCs, I can not find the reference now though.

To sum it up, the combing is not strange on its own - it is a fact of life as soon as "you" start mathematically manipulating the data. The mystery is the combing seen on different models (confirmed personally by me on 350D, 30D and 5D) in JUST A FEW photos taken at FULL STOP ISO settings when most other photos have perfectly smooth histograms.

Alan T. Price
September 6th, 2006, 04:37 AM
The mystery is the combing seen on different models (confirmed personally by me on 350D, 30D and 5D) in JUST A FEW photos taken at FULL STOP ISO settings when most other photos have perfectly smooth histograms.

In your original post you showed the histograms for two full stop ISO 100 images but one had a third stop compensation, making it in effect a non-full-stop ISO 125. Are there other examples with full stop ISO and no EC that have combing ?

Peter Ruevski
September 6th, 2006, 07:48 AM
In your original post you showed the histograms for two full stop ISO 100 images but one had a third stop compensation, making it in effect a non-full-stop ISO 125. Are there other examples with full stop ISO and no EC that have combing ?
This is a very good point and exposure compensation (EC) being handled as (1/3) ISO change was my first guess too. But now I am convinced that it is not the cause. Here are my reasons:

1. There are combed photos from my 30D that were shot with EC=0 I do not have any posted on the site though.

2. There are combed photos from my 30D that were shot in manual mode

3. All of the combed photos from my friends 350D are shot in manual mode

4. The 5D combed photos that Mike Robbins gave me are shot with EC=0

5. The combing in intermediate ISO photos is much mode severe (dense) than the "mysterious" combing. Here is an example at ISO 125:
http://www.pages.drexel.edu/~par24/rawhistogram/IMG_1754.png
The graph looks solid because of the dense combing. Here is a zoomed in section:
http://www.pages.drexel.edu/~par24/rawhistogram/IMG_1754_zoom.png

6. And last but most important - if you point the camera to a constantly illuminated scene on a tripod in Av or Tv mode and change the exposure compensation you will see that the aperture or shutter speed respectively will be adjusted accordingly. If the ISO were adjusted instead they would have stayed constant - otherwise an over/under exposure would occur.

Of cource it is seems that sometimes the camera is doing some very slight scaling for some reason, which causes the "mysterious" combing... but why?... and when?...

Alan T. Price
September 7th, 2006, 11:57 AM
well Peter I can't think of anything else to help out, except one last desperate idea....

Could there be any EC applied to raw data in a program such as DPP, and could your reading of the raw data be influenced by that EC ? This would depend, I assume, on whether you are reading the data directly or by using a Canon-supplied routine that presents the data to your program.


Also, have you zoomed in on the combing close enough to determine how many values are being sacrificed and whether or not they are consistent in some way. e.g. is it every 4th value, or every other odd value, etc. There are twice as many green pixels as red or blue. Are both sets of green losing the same values ? May not help at all but this has now got to the point where it's just fascinating.

John Sheehy
September 18th, 2006, 09:32 PM
Hi, remember me?

So I presume when you said "all ISO settings" you included intermedate 1/3 stop ones like ISO 125? If so and you did not see any combing then the 5D would certainly seem to have "proper" i.e. alanog amplifier driven intermediate ISO settings.

Yes and no. There are other ways of obtaining non-optimum ISOs.

I started another thread about the 30D intermediate ISOs after you stopped visiting DPReview (or you just missed that thread), but someone gave me 5D blackframe files to look at, and I found a problem with the noise levels, that made me recant my earlier declaration that 5D had amplified, real, intermediate ISOs. Amplified they seem to be, but they are still not as real as the 100/200/.../1600 series. The histograms look good at all ISOs (except that every 15.5th value is unrepresented, in the samples I received), but the actual noise values for the 5D did no smoothly graduate from 100 to 1600; they are, in fact, separately amplified before going into the A2D converter, and therefore have more blackframe noise than they should at the intermediate ISOs.

The pattern is this:

100 2.0
125 2.4
160 2.9

200 2.0
250 2.5
320 3.0

400 2.2
500 2.7
640 3.3

800 3.0
1000 3.7
1250 4.6

1600 4.6

Horizontal banding noise is directly proportional to the numbers within each triplet (but not the whole list); vertical banding noise is not (but vertical banding noise is rarely visible). These are the numbers that determine dynamic range; 160 has nearly as much blackframe noise as ISO 800.

Of course, blackframe noise is the "noise floor" that affects DR and deep shadows mostly, and the noise in the highlights and the midtones (and the darker midtones at higher ISOs) is photon shot noise, which is affected by photon collection in the sensor wells and has nothing to do with ISO settings per se (except as they affect metering, and therefore, exposure).

Conclusion: Don't use these extra 5D ISOs; underexpose at the lower base ISO, and get the extra headroom (or expose all the way to the right and get a better capture).

John Sheehy
September 18th, 2006, 09:49 PM
Just a thought.

Lots of interesting ideas there, but an important point missed is that what we call "real" ISOs on the 30D are heavily optimized - ISO 1600 only has 2.26x the readout (blackframe) noise of ISO 100, even though it amplifies 16x as much. ISO 400 only has 1.18x the readout noise as ISO 100, even though it amplifies 4x as much. 1600 has 1.19x as much as 400, with 4x the amplification. The higher ISOs are highly optimized to avoid readout noise proportional to amplification. When the 30D decides to create its ISO 125 by digitizing a shot metered for 125 and amplified for 100, it multiplies the readout noise by exactly the amount of the "amplification", which is totally non-optimal in the trend set by the 100/200/.../1600 group. In fact, the readout noise of 125 on the 30D is about the same as the readout noise at 640, because 640 has its noise *divided* from the amount at ISO 800.

John Sheehy
September 18th, 2006, 09:51 PM
I have used the IRIS program and looked at ~200 cr2 files with iso's from 100 - 1600. No combing artifacts at all. Will run a better (more controlled) test this weekend.

Did you try zooming into the histogram so that it is only a couple hundred RAW values wide, and widen it on the desktop?

Asher Kelman
September 18th, 2006, 10:18 PM
Hi, remember me?



Yes and no. There are other ways of obtaining non-optimum ISOs.

I started another thread about the 30D intermediate ISOs after you stopped visiting DPReview (or you just missed that thread), but someone gave me 5D blackframe files to look at, and I found a problem with the noise levels, that made me recant my earlier declaration that 5D had amplified, real, intermediate ISOs. Amplified they seem to be, but they are still not as real as the 100/200/.../1600 series. The histograms look good at all ISOs (except that every 15.5th value is unrepresented, in the samples I received), but the actual noise values for the 5D did no smoothly graduate from 100 to 1600; they are, in fact, separately amplified before going into the A2D converter, and therefore have more blackframe noise than they should at the intermediate ISOs.

The pattern is this:

100 2.0
125 2.4
160 2.9

200 2.0
250 2.5
320 3.0

400 2.2
500 2.7
640 3.3

800 3.0
1000 3.7
1250 4.6

1600 4.6

Horizontal banding noise is directly proportional to the numbers within each triplet (but not the whole list); vertical banding noise is not (but vertical banding noise is rarely visible). These are the numbers that determine dynamic range; 160 has nearly as much blackframe noise as ISO 800.

Of course, blackframe noise is the "noise floor" that affects DR and deep shadows mostly, and the noise in the highlights and the midtones (and the darker midtones at higher ISOs) is photon shot noise, which is affected by photon collection in the sensor wells and has nothing to do with ISO settings per se (except as they affect metering, and therefore, exposure).

Conclusion: Don't use these extra 5D ISOs; underexpose at the lower base ISO, and get the extra headroom (or expose all the way to the right and get a better capture).

In the noise measurements shown, could you explain how these are arrived at? You may well have mentioned that in a previous post. Is there a PC/Mac utility to measure this?

The points you make are very important. I have always thought I was doing myself good to shoot at 160 rather htan 200 ISO!

I'd like to see some examples of the type of image where either the noise would have an effect on the printed image or where the combing of every 15.5th value in the histogram might degrade the final image too.. (BTW, What do you use to demonstrate this?).

Asher

John Sheehy
September 18th, 2006, 10:45 PM
In the noise measurements shown, could you explain how these are arrived at? You may well have mentioned that in a previous post. Is there a PC/Mac utility to measure this?

That was my first post here, but you probably recognize my name from the old RG forums. The PC program IRIS http://www.astrosurf.com/buil can load the RAW blackframe, and then issuing the "stat" command in the console window gives the noise ("sigma").

The points you make are very important. I have always thought I was doing myself good to shoot at 160 rather than 200 ISO!

If the scene is high-key, then you probably are, as shot noise will be the main noise, and there will be less of it metering for 160. If you are concerned with DR, though, 160 is not the place to do it.

What I am saying is that there isn't any monolithic "noise" in an image; there are two basic groups of independent noise; noise that is related to the signal (such as shot noise) and noise that is a blanket over the entire image (readout, or blackframe noise). The former is directly related to real, analog exposure. The latter is dependent upon the ISO setting, also.

I'd like to see some examples of the type of image where either the noise would have an effect on the printed image or where the combing of every 15.5th value in the histogram might degrade the final image too.. (BTW, What do you use to demonstrate this?).


Seeing combing at that level will be very difficult. I'm not saying it's a big problem. In the camera I received the files form, RAW values 107, 123, 138, and 154 were grossly under-represented (the few that were there may be the result of interpolation for mapped-out pixels). Black is 128 in recent Canon cameras. It would take a very slow gradient in the deep shadows at a very low ISO to see the effect. The combing itself is not a big problem, as there is too much noise in all cameras at all ISOs for minor posterization to jump out at you. The posterization is a symptom, though, of some unnecessary math going on; math that should not be done at all IMO; I believe that digitized number should be stated literally in the RAW data, and that all scaling should be done by the converters, by stating figures in RAW metadata, or by external knowledge. That's how the 10D differed it's 3200 from 1600; the RAW levels were proportional to absolute exposure; IOW, turning the ISO from 1600 to 3200 on the 10D in manual mode did not affect the RAW levels; the converters knew to scale the 3200 data by +1 stop more than 1600 data in the conversion, and it worked. Why scale anything, and clip away highlights, or turn two RAW values irreversibly into one?

As far as the increased noise is concerned at ISOs like 160, try shooting the same scene with deep shadows at 160 and 200 (with the same EC), and bring the shadows up and see what they look like. I'm sure you'll see bolder noise and banding in the 160.

John Sheehy
September 18th, 2006, 10:53 PM
Imagine a simple monochrome image, which only has pixels with five shades of gray 0, 1, 2, 3 and 4. If you multiply every pixel by 2 now your image will be "twice as bright" (in fact it will have twice the dynamic range).

The noise floor doubles in value, too, so there is no gain in DR.

John Sheehy
September 19th, 2006, 06:32 AM
I have do the test at all iso settings from 50 to 3200 and had no combing artifacts at all! That surprised me because I expected to see them at 50 and 3200 ISO!

You might want to try blackframes (short exposures with the lens cap on) at ISO 1600 and 3200 on the 5d, load them into IRIS, run the command console and type "stat" for each. If the 3200 "sigma" value is 2x the 1600 value, then the camera is probably using a second, low gain amplifier following the optimized 100/200/.../1600 amplifier system. Perhaps it is applying 1.25x gain for the 125/250/... group, 1.6x for the 160/320/... group, and 2x for 3200.

John Sheehy
September 19th, 2006, 06:45 AM
Hi Richard, The ISO 3200 lacking combing does not surprise me – it is well established that ISO 3200 a.k.a. "H" is simply ISO 1600 pushed (i.e. underexposed) one stop and a flag set in the file telling the raw conversion software to do +1 stop exposure adjustment. As far as the actual raw data is concerned ISO3200 is exactly the same as ISO1600 - that is why people say it is not useful when shooting raw. When shooting JPEG the firmware does the pushing so it is useful.

For what camera? The 10D does this, and I assume the 300D hack. The 20D does not do it that way; the 20D actually doubles all the RAW values for 3200, clipping away a stop of highlights that would be saved if it were done the way you suggest, which, I believe, is the way it should be done, especially when you consider the fact that many scenes that call for exposure indices like 3200 have light sources within the scenes, and need the expanded headroom. Canon seems to be more interested in maintaining a consistent relationship between RAW levels and relative exposure levels, to the detriment of the quality or usefulness of the data. It would explain every quirk we're discussing here - the light combing at ISOs 100/200 etc could just be the camera saying "I think the amplification was only 96% of what it should have been, so I will scale the digitized data", rather than just leaving it alone.

John Sheehy
September 19th, 2006, 06:52 AM
Your impression is correct - 100%. The combing is different though - e.g. ISO125 is “good old down to zero” combing as the examples I posted earlier. ISO160 however is a "half hearted not quite down to zeroes" kind of combing

The 160 group's artifacts in the histogram are the high spikes, not the low ones. The high spikes are two digitized values becoming one.

Peter Ruevski
September 20th, 2006, 09:15 AM
Hi, remember me?

Hi John,

Of course I remember you. Your early posts on the intermediate ISOs at dpreview are what started me "digging in raw data" :-)

Unfortunately lately I have been quite busy and have visited this forum only occasionally and dpreview not at all. So I have missed your thread, but I will go back and look for it when I find some time.

As usual you have made some very good points and I will consider them carefully and write a single reply - hopefully sometime this week, maybe next.

Best regards,
Peter

Adrian Warren
October 3rd, 2006, 02:42 PM
As a matter of interest, are the shots exhibiting combing in focus? Does the combing appear on entirely OOF shots?

I'm wondering about Airy Discs:
http://en.wikipedia.org/wiki/Airy_disc

John Sheehy
October 3rd, 2006, 03:52 PM
As a matter of interest, are the shots exhibiting combing in focus? Does the combing appear on entirely OOF shots?

I'm wondering about Airy Discs:
http://en.wikipedia.org/wiki/Airy_disc

No. You do not see the detail of the airy disks, as they are broken up by the geometry of sampling, bayer CFAs and the AA filters, and even if you could they would still be analog, both individually, and as a collection of millions of diffracted points of light.

The combing spoken of here is a simple multiplication/division with whole numbers, performed by software/firmware or circuitry; such as the numbers:

0 1 2 3 4 5 6 7 8 9

becoming:

0 1 2 4 5 6 8 9 10 12

rather than:

0.00 1.33 2.67 4.00 5.33 6.67 8 9.33 10.67 12.00

John Sheehy
October 9th, 2006, 08:24 PM
Hi John,

Of course I remember you. Your early posts on the intermediate ISOs at dpreview are what started me "digging in raw data" :-)


It gets worse. I am *really* starting to get very annoyed with Canon. My 20D is giving me trouble, and I wanted an EF-S backup/second camera, anyway, so I bought a 400D. First thing I did when I got home with it was to test the noise levels, and RAW saturation levels. ISO 100 and 200 had less noise relative to RAW saturation than any Canons other than the 1*mkII series. After taking a number of shots, I loaded one ISO 100 image into IRIS, and immediately I noticed that there were pixels registering 4095. That should be good, but it is not the same as the other times I looked at clipped ISO 100 data (3726, or something like that was the clipping point at ISO 100). Then, I looked at a histogram of the RAW file, and alas, the camera multiplied everything by 31/25! That's a loss of about .3 stops of highlights, because some Canon engineer thought that it might be important, for some unknown reason, to scale the original RAW capture, as opposed to simply embedding a unique whitepoint value in the RAW file's metadata.

Canon, stop screwing with *MY* negatives. Now I can't even assume I know how much DR the camera has, as it may throw away .3 stops of highlights (and DR) with a seeming roll of dice, forcing me to expose lower and get more noise.

What in the world is Canon thinking with this nonsense? Canon, please leave the RAW data alone. The way it works in a 20D is a beautiful thing. 128 for blackpoint at every ISO, and values up to 4095 for every pixel. The 30D introduced inter-math based extra ISOs - yuck. It also scales the RAW data with integer math on occasion for the base ISOs as well, mildly in the 30D, and now sickly in the 400D.

Leave it alone, Canon. Just put a different whitepoint in the metadata, to instruct the converters to scale values. LEAVE THE RAW NUMBERS ALONE.

You thought your 27/26 scaling was bad!

Jason C Doss
October 10th, 2006, 06:52 AM
Have you looked at the Fuji DSLRs? I'm thinking of switching to the S5 when it comes out.

Frank Doorhof
October 10th, 2006, 12:29 PM
I will get burned/flamed for this......

I know it's always better to do something else, but why not just do what the camera is good at, taking pictures.

I'm a perfectionist when it comes to my photos but I look at the endresult and work with that, in other words, I learn to use the tool which is the camera.

The noise levels on the 5D are so wickedly low that even prints on A4 format from ISO3200 are almost clean, to be honest I couldn't care too much for numbers, the end result is what counts.

Highlights are a problem with digital, that is known for a long time, fuji is better with highlights thanks to the design of the censor, also that is known, but when I look at the sharpness and depth/color of the endresults I want my Canon :D

John Sheehy
October 10th, 2006, 05:47 PM
I'm a perfectionist when it comes to my photos but I look at the endresult and work with that, in other words, I learn to use the tool which is the camera.

The digital negative is mine. It should just be a digitization of the amplified sensor readout. Canon is compromising the integrity of my negative for no good reason. There is absolutely no need to scale RAW data after digitizing it; if the camera thinks that the amplifier didn't achieve the target amplification, it should just put a metadata in the RAW file that says that the data needs to be scaled by 1.1923 (31/25), or state that the whitepoint is 0.8387x (25/31) what it normally is. What they do, instead, is make the data less accurate and throw away dynamic range. I don't see any value in being complacent about that.

One reason I didn't even consider the 30D was the make-believe ISOs it has, and the fear of accidentally using them, or having to dial through them, to get to the ones I want. I thought that since the 400D didn't have these extra ISOs, it would be straight-forward with the RAW data, avoiding the nonsense.

The noise levels on the 5D are so wickedly low that even prints on A4 format from ISO3200are almost clean, to be honest I couldn't care too much for numbers, the end result is what counts.

Like it or not, your image is recorded and developed as numbers. Better care should be taken of those numbers, IMO.

The noise levels on the 5D are nothing special, compared to other Canons at the pixel level. The main benefit is that there are 12MP, and that allows more resolution, and the fairly large pixels each have potentially better contrast with their neighbors, because the pixel pitch is less demanding on the MTF of the optics. The 5D exhibits posterization gaps in its data, too; at least some copies. It is inexcusable; it is just a horrible way to deal with what may or may not be a problem, however subtle. It is not a sacrifice for something better; it is just needless destruction. The person who made this decision is not a good steward of data. There are better ways to deal with the issues, real or imagined.

Highlights are a problem with digital, that is known for a long time,

This is a different problem. The dynamic range is *changing* from frame to frame at ISO 100 on my 400D, by 0.3 stops. That means that assuming that the higher range is available will be risky when the occasional lower-DR frame happens.

fuji is better with highlights thanks to the design of the censor, also that is known, but when I look at the sharpness and depth/color of the endresults I want my Canon :D

The Fuji method of increasing DR does nothing to improve sensitivity; it simply de-sensitizes half of the pixels so they don't blow as easily. The "shadows" of the less-sensitive sensels add noise if they are used for increasing resolution of the overlapping ranges. I haven't seen the Fuji software, but if I were writing it, I would provide a slider that acted like a crossover frequency in audio, only as a crossover point for the light levels to fade the two images for the overlapping areas. It might be better to leave out the extra resolution if it increases noise.

I want my future increases in DR to come from the shadow end of things. I already have ND filters.

Peter Ruevski
October 11th, 2006, 04:47 PM
I will get burned/flamed for this......
Not at all - in fact your post made me think why some people (including myself) pay so much attention to the technical details of photography sometimes. After thinking about it a bit I think I can answer for myself:

I know it's always better to do something else, but why not just do what the camera is good at, taking pictures.
This is the crucial question. And in a long and indirect way it leads to the answer of my own question above.
Taking pictures – for me - has always been easy, pleasant and a lot of fun – I have been doing it since I was six. Buy a roll of film, shoot, take it to the lab, get the (black and white) prints, enjoy - nice and simple.
Then in high school I started to develop and print my own black and white photos. Initially I was learning and was happy to just get a decent looking recognizable photo. But as I learned I started to see that a photo is not an "absolute truth" – something which strangely enough I had not realized until then – and I was quite "shocked". I started playing with different contrast papers and developers, burning, dodging and… something for which I do not know the proper term in English – when you expose the paper to white light for a short time half way through development. I would spend inordinate amounts of time (nights) and paper printing the same photo again and again trying different things.
So now the look of a print had become to a large extent a matter of my interpretation and while I found that slightly disconcerting I actually enjoyed it.
I certainly spent much more time in the dark kitchen at night then I spent shooting :-)

Then color film became cheap enough (I am not a hundred years old – I simply lived in Eastern Europe:-) and I started shooting only color. The dark room equipment started collecting dust in the wardrobe and photography was once again an absolute thing for me. Buy a roll, shoot, take it to the (cheapest possible) lab, get prints... there was no accessible/cheap pro lab and therefore there was no custom printing or "I do not like these prints" for me. The operator of the minilab made all the decisions for me and in fact I kind of liked it, my mother certainly liked it – the kitchen was once again safe at night :-)

... a few years pass ...

Which brings me to May of this year when I eventually bought a 30D. Being a programmer I did not even for a second consider shooting JPEG – it is an ATROCIOUS destruction of data - and here I totally agree with John! I did shoot RAW+JPEG though and for a whole month did not touch the RAW files. But then of course I went back to the "darkroom" or rather lightroom (is this trademarked by Adobe already?) and I have to say having color – at least for me – makes it much harder. Now the look of my photos is even more a matter of my interpretation – and I still have not gotten used to it.

To finally answer your question "why not just do what the camera is good at, taking pictures" - I am, and since that is as easy and pleasant as it has always been I do not need to discuss it in forums :-) The artistic side of developing RAW files on the other hand is still too hard for me and I do not consider myself qualified to discuss it – for now I am still reading and learning.

But treating RAW data as numbers (and it is numbers!) and analyzing it in every possible way - I am very qualified to do - and feel confident to comment on what I find. Therefore when I found something interesting I decided to share it.

As to why the combing is bad I think John answered very-very well in his last two messages.

Frank Doorhof
October 11th, 2006, 05:13 PM
I understand :D
And I love in depth reviews and discussions but I just worked my way to some threads on DPR where people are bashing cameras on technical things only, which normally is no problem at all.

That's why I wanted another view :D