Combing artifacts in 30D RAW histograms - what casues them? (Chuck Westfall?)

[Peter Ruevski]

Hi all,

It has been two and a half months since I bought a 30D (after 14 years with two film SLRs) - it is a wonderful camera!

Being a programmer however I just had to look into the RAW data that the camera generates. One of the reasons I did it is because I had read that the intermediate ISO sensitivities (e.g. ISO 125) on the 30D are achieved by scaling the already digitized signal in the firmware. It indeed turned out to be the case. As a result the RAW histograms of photos taken at intermediate ISOs have quite heavy (dense) combing artifacts – which is not at all surprising.

However I also discovered that the RAW histograms of some photos taken at full stop ISO sensitivities are also combed! Here is an example of a good histogram:

ISO 100, EC 0, 1/80 f/22


And here is and example of a combed one:

ISO 100, EC +1/3, 1/500 f2


What the above shows is that even at full ISOs the firmware of the 30D sometimes manipulates the RAW data coming from the A/D converters before writing it to the CR2 file – so the “RAW file” is not really RAW in these cases.

How often does it happen? Well I wrote a program (and a batch file) that can go through any number of CR2 files and check whether their histograms are combed. Using that found that from the 1304 CR2 files that I have – all of them shot at full ISOs - 216 have combed histograms. The period of the combing pattern varies from one photo to the other. I have put together a web page that has many more examples, the software used and some examples from other Canon dSLRs.
http://www.pages.drexel.edu/~par24/r...histogram.html

Even though I tried – I was unable to find any correlation between the shooting conditions and the combing appearing in the histograms. I find all of the above very interesting and would really like to hear any ideas or explanations of why it is happening. You can also try it on your own RAW files using the software I wrote.

Best regards,
Peter Ruevski

[Chuck Westfall(Canon USA)]

Hi, Peter:

I appreciate your interest, but we do not comment on our image processing algorithms. Our cameras are basically offered "as is," and we do our best to make sure that image quality is as high as it can be. Canon's official statement on EOS 30D image quality is as follows:

"The image quality of the EOS 30D and all other EOS Digital SLRs conforms with Canon's internal quality standard at all available ISO speed settings. We have no further comments to offer on this issue."

Best Regards,

Chuck Westfall
Director/Media & Customer Relationship
Camera Marketing Group/Canon U.S.A., Inc.

[Asher Kelman]

Originally Posted by Chuck Westfall(Canon USA)

Hi, Peter:

I appreciate your interest, but we do not comment on our image processing algorithms. Our cameras are basically offered "as is," and we do our best to make sure that image quality is as high as it can be. Canon's official statement on EOS 30D image quality is as follows:

"The image quality of the EOS 30D and all other EOS Digital SLRs conforms with Canon's internal quality standard at all available ISO speed settings. We have no further comments to offer on this issue."

Best Regards,

Chuck Westfall
Director/Media & Customer Relationship
Camera Marketing Group/Canon U.S.A., Inc.

Chuck,

Again, we appreciate your following threads here for us!

Could you state whether or not the 5D or 1D series share identical ways of dealing with luminance? IOW, can we avoid such 30D effects by buying a more professional camera or do all the cameras use the same basic translation mathematics?
Asher

[Peter Ruevski]

Hi Chuck,

Thank you very much for your quick reply and for following user forums.
As for the combing artifacts in round ISO speed RAW histograms - I do not think that they visibly affect image quality. In some almost identical photos where one had the combing and the other did not - I could not see a difference in the image quality. In fact I am very satisfied with the image quality of my 30D! What is more, most post-processing operations do "much worse" things to the data.

So this is not about image quality at all - and I was very careful not to imply such a thing - it is more of an "academic" interest for me - an unsolved mystery in a way. Canon's "no comment" policy is perfectly understandable, I just though no harm can come from asking :-)

As for the intermediate ISOs - I do not mind them at all - and they are not a "mystery". What is more they would certainly be useful for people shooting JPEG - and when shooting RAW one can achieve the effect "consciously" in RAW conversion and just avoid them.

Thank you again for your reply.

Best regards,
Peter Ruevski

P.S. Everyone else - the mystery remains :-)

[Chuck Westfall(Canon USA)]

>>Could you state whether or not the 5D or 1D series share identical ways of dealing with luminance? IOW, can we avoid such 30D effects by buying a more professional camera or do all the cameras use the same basic translation mathematics?<<

Hi, Asher:

Those questions are precisely the ones that would lead to 'further comments' on image quality that Canon Inc. is not willing to discuss. Sorry!

Best Regards,

Chuck Westfall
Director/Media & Customer Relationship
Camera Marketing Group/Canon U.S.A., Inc.

[Stan Jirman]

Maybe Peter can run his batch program on a series of CR2 files from a 5D and some 1D-series cameras? Then we won't have an official Canon statement, but empirical evidence.

[Harvey Moore]

It appears to me that you are analysing dpraw interpretation of the files, not the actual raw data.

[Asher Kelman]

Originally Posted by Stan Jirman

Maybe Peter can run his batch program on a series of CR2 files from a 5D and some 1D-series cameras? Then we won't have an official Canon statement, but empirical evidence.

Stan, I was thinking the same thing. I do check my histiograms and I don't recall seeing combs on my 5D images. I'll look out for them in my current processing.

Peter, how many files would you need? Which ones?

Asher

[Nill Toulme]

My grandfather always used to say "Never trust a man with comb tracks." Is that what y'all are talking about?

Nill
~~
www.toulme.net

[Peter Ruevski]

@Harvey Moore

Indeed I am analyzing the output of dcraw, however I am using it in the mode in which it outputs unmodified RAW data (not even Bayer demosaicing is done) to a 16bit PGM file. I posted a similar thread on dpreview (http://forums.dpreview.com/forums/re...ssage=19009766) and a similar question came up. I have traced through (debugged) the dcraw code path that outputs the RAW data and I believe that it is indeed untouched 12bit raw data as it came from the ADCs stored in 16bit values inside the PGM.

Bottom line - I am 99.9% certain that the combing is not introduced by dcraw.

@Asher Kelman

I would love to analyze some 5D files. Ideally it would be the same scene shot under different ISO settings - including 1/3 steps - and preferably one that you would not mind me posing the pictures of on the page I mentioned in the original post.

Keep in mind however that the combing may be a rare occurrence - I have a friend with a 350D, he ran the search on about 500 of his CR2 files and only two showed combing.

If you are using a PC (Windows) you can PM me and I can send you the required files and instructions so you can run the search on a big collection of files.

As for seeing the combing in the histograms - I am talking about histograms of the RAW data - even processing the RAW file to a 16 bit linear TIFF would already hide the effect - the bayer demosaicing is in effect an interpolation. The only other program that I am aware of (apart from mine) that can show the real RAW histogram is Iris http://www.astrosurf.com/buil/us/iris/iris.htm - and it shows the combing too. See this (http://forums.dpreview.com/forums/re...ssage=19131963) for an explanation how to use it - it has a GUI so it is easier to use (and immensely more powerful) than my program. The only advantage of mine is that it can be used to do an automated search through many files.