Quality Control: My 1D MkIIN Experience

Hello,

I have been having endless hassles with my Mk II N, it's been to and fro the Canon service centre several times now. I am in South Africa, and the only authorised repair centre practically available to me is Camera Tek

When I got my camera in October, it's always had an alarming amount of hot pixels. Every time I send it in to get them mapped out, I either end up with more of them, or no change. They eventually conceded to replace the CMOS sensor - they had my camera for four weeks, and though they gave my a loaner (also a MkIIN) this camera itself had a somewhat faulty sensor - all photos looked like they were taken on rough canvas, or badly scanned (small square patterns) even at ISO320. I put up with this, with the hope that they'll finally sort my camera.

When I eventually got my camera back, it was a complete disaster (I'll post sample images later). All images now have very, very pertinent pink noise splattered over them. A 1s exposure at ISO1600 with the lens cap on actually turns out pink/reddish - it couldn't even nearly be called anything approaching black. In other words, my camera is completely faulty. Ironically, a snapshot taken at ISO200 looks OK (at a glance, that is).

So, after having my camera for 4 weeks, they did not bother to test it once at any higher ISO, and certainly not under the test condition which I brought it in for in the first place - ISO1600 at 1s with the lens cap on.

I e-mailed them again (and will get on the phone with them once the main technician is available), but I wanted to know if anybody else has had experience like this with a 1-series camera? And what can I do? I am now expecting them to replace my brand new camera (which I now consider completely faulty) - I don't even want them pulling it apart any further to try and solve the problem. I hope this is reasonable, this saga has been going on for almost 4 months now.

Sample Image

Hello, the camera is in to Camera Tek again, and I've managed to upload a sample image. First of all, the head technician does claim to have tested the camera with a 1-second exposure, so I guess something must have gone wrong in the hours that elapsed since his, and my own, testing. This looks very much like thermal noise to me.

They sent a driver to come and pick it up ASAP, and are very courteous about the whole matter, so I cannot fault them, I'm still just a bit disappointed about this happening straight after receiving the camera after a 4-week repair (they apparently waited for parts). It also has a nasty cluster of 4 bright red pixels in the middle of the frame (not very visible in my scaled sample image).

Here is a 1-second exposure, at ISO3200 (straight RAW conversion in Apple Aperture, no adjustments, 1024x JPEG export). Though quite a bit of noise is to be expected at this sensitivity level, I certainly expect an overall black picture. I'll let you know what their findings are - any preliminary theories from readers?

Dawid Loubser said:

... any preliminary theories from readers?

Click to expand...

It looks like "Amp Glow", but I don't know the details about shooting conditions and postprocessing.

An analysis would require at least 2 Raw files taken some 6 seconds apart (allowing the previous image to be written to the card and the LCD to switch off) with viewfinder and lens covered. Long exposure noise reduction should be switched off, just in case. Repeating the experiment with the camera acclimatized at different ambient temperatures and at the beginning/end of a shooting sequence is needed to see if the phenomenon gets worse with increasing body/amplifier temperature. Longer exposure times would also cause an increase, just like long LCD viewing may have an effect.

Bart

Bart_van_der_Wolf said:

It looks like "Amp Glow"...

Click to expand...

Amp Glow is starting to sound very plausible indeed, the more I think about it. For a similar exposure time, the effect increases dramatically as ISO increases, and this directly cranks up the amplification. I have done further testing, and images taken (ISO1600, 1s, body cap + eyepiece shutter closed)

• 1 minute
• 10 minutes
• 1 day

apart all look near identical. I have not done anything to test temperature-related effects, but I'm sure the service centre will do proper, detailed testing in anyway.

Bart_van_der_Wolf said:

I don't know the details about shooting conditions and postprocessing.

Click to expand...

The image has absolutely no post-processing applied, the only changes are colour space (I shoot in Adobe RGB all the time, exported as sRGB for web viewing) and down-res.

Thank you for your input!

Dawid Loubser said:

Here is a 1-second exposure, at ISO3200 (straight RAW conversion in Apple Aperture, no adjustments, 1024x JPEG export). Though quite a bit of noise is to be expected at this sensitivity level, I certainly expect an overall black picture. I'll let you know what their findings are - any preliminary theories from readers?

Click to expand...

That's pretty much what a high-ISO blackframe looks like from some converters, if you push the exposure. The natural whitebalance of digital cameras is not daylight; not even close. White light records as a cyanish-green in the RAW data, and daylight white-balance in the converter boosts the red by about a stop, and the blue by about a half-stop, resulting in your magenta cast in the noise (and there is always noise in a lens-cap exposure, even one of 1/8000s).

I have to wonder why the top and right edges are darker; though. Is this a true blackframe?

Try 1/8000 in a closet.

John, thank you for your input, but forgive me if I assume it's somewhat flawed:

• I specifically stated that no adjustments were made, yet you assume I pushed the exposure, or don't understand my RAW converter
• I am intimately familiar with the noise characteristics of this camera model
• This is a true blackframe; taken with body cap on, and eyepiece shutter closed

No, this is clearly a faulty camera.

Problem Solved

Hello,

I am happy to report that the Canon Service Centre (Camera Tek) very promptly assisted and solved my problem: They sent a driver over to come and pick up the camera at noon, and by the end of the day, I had my camera back, which now renders an amazingly smooth ISO3200 1-second darkframe.

Taken under exactly the same circumstances (including closely matching temperature), compare this to my original posting: Please note that for this, and the previous image, all noise reduction in my RAW converter has been disabled (Apple Aperture). Oh yes, and who was that alluding to user error?



Now for the synopsis: The problem was indeed the sensor. After they replaced the first sensor on Friday, they tested it, and it was fine. However, as soon as I started using it hours later, something must have blown (and it is likely related to the amplifier), as since then, the sensor output was unusable. Replacing the sensor board (which apparently also contains the amplifier) again now leaves me with a very happy camera, this is the first time ever with this camera which I get not a single problematic hot pixel (or cluster). I can't wait to take some good low-light shots again.

They have sent the sensor in to Canon for testing, and will let me know the outcome (what went wrong). I'll then also post it here for brevity. So... kudos for excellent customer service by Camera Tek, but bad for canon quality control, with two dud sensors in a row...

Correction

Dawid Loubser said:

...after having my camera for 4 weeks, they did not bother to test it

Click to expand...

In all fairness, I have to withdraw this harsh statement (I should have qualified it with "seemingly"), as they showed me proof (test images). What happened instead was the unlikely scenario of something going wrong between them testing it, and me testing it - a few hours apart.

Its good to see that Canon finally found and resolved the problem, we all know how frustrating things can get when trying to get results.

Problem NOT Solved, it seems

Will you believe it, the problem is back... *sigh*. It was, in fact, not solved, and looks identical now to my first post! So there was indeed likely nothing wrong with the CMOS sensor, and the problem is obviously caused by something else. Still looks like severe amp glow to me.

What's interesting, is that the same identical hot pixels appear after a CMOS sensor change, so the hot pixels (not to mention the terrible purple glow) is caused by something else.

100% Center Crop, ISO3200, 0.5s - 1.0s, dark frame


This is getting very frustrating...

Dawid,

Just checked my 1DMkIIN and similarly 1 red pixel and 1 green pixel showed up momentarily in the RAW files in identical places and then they disapeared in both Lightroom and Silkypix, when the render engines kicked in.

I wouldn't worry about it too much.

Incidently, are you shooting in RAW?, what application are you using to convert?

Hi Dawid,

Is it a case you are opening the image in cs2 as pro-photo, but image originally in argb colour space? I get that (Lieca purple look) if I expose as you have done with my 20d.

Best wishes,

Ray

Hi John,

Seems you're playing too - in dpp everything black, in cs2 jpgs black, but raws are noise. I've not checked if that is general for all cs2 settings. argb/srgb/raw etc. colour spaces.

Best wishes,

Ray

No, no, no...

This is not a problem with my RAW workflow, I know what I'm doing in that regard. Just to put everything into context, THIS is what an identical (1s, ISO1600, same conditions) dark frame exposure looked like before they replaced my sensor the first time. If you look at that image (full size JPEG, slightly compressed) you'll see it has a ridiculous amount of hot pixels in it - not one or two or ten, but hundreds, which prompted my to send my camera in in the first place. But it's overall black, as expected.

Furthermore, updating the firmware to have the camera map them out did in fact not work (the camera "ignored" their "image file") which caused the whole CMOS replacement...

The pink is not a colour space or RAW workflow problem, something's wrong with the camera hardware. If this was a workflow problem, I'd not see the same thing on the camera LCD.

I use Apple Aperture, which I have been using since the first version, and I have been getting great results with it (see my gallery on DeviantArt.)

For some reason, though I posted here for technical help / advice, nobody seems to "believe" it's the camera hardware? Has nobody seen anything like this on a 1DMkII(N) before?

Thanks for your input.

John_Nevill said:

1 red pixel and 1 green pixel showed up momentarily in the RAW files

Click to expand...

This I could handle - I'm no pixel peeper. 10 I could handle. But I have hundreds! In clusters of 9 bright red pixels each.
Oh, and I use Apple Aperture most of the time. But the same pixels show up in Canon DPP, so it's not a tools or workflow issue...

Hi Dawid,

It is not a question of doubting it's a hardware problem, just my attempt to replicate something like it here. I would suggest you take aperture out of the loop in any testing that you may be doing, as I do not think canon supports third party software. In DPP, if you move the sliders around, does it get near black in any position? Try it with new RAW images, discard the previous, which may have inadvertently been altered.

It is odd that the camera appears to work for you initially, then after an hour or so of your testing, it fails. Changing the sensor has apparently had no effect - both your images show the same patterns.

If I was as confident as you are that it is only a hardware problem, then I would be looking for a complete replacement.

Best wishes,

Ray

Dawid, Can you post a RAW image with this problem?

I agree that it's odd that initial testing looked fine, and from there onward it's disastrous. The technician's theory was that some or other component fails quite soon, maybe caused by an overload of some sort due to another faulty component. I can't explain it any better than what the Canon technician can - as I said, the camera is going in today, so hopefully he can have a look tomorrow. He did confirm that the sensor was replaced (after I questioned him about the hot pixels remaining in the same place pre/post the change) and he is doing to tart replacing image processing / DC boards.

I am however absolutely certain that there is not even the remotest possibility of this being a software or workflow problem. I have been a canon RAW shooter for a long time, and I am a specialist software engineer (my day job). I believe I know my software, I know my workflow, and (most important of all) I know my camera. *And* this shows up on both RAW and JPEG.

Furthermore, these is no possibility of me inadvertently mis-adjusting my RAW files, as I follow a completely non-destructive workflow - I never touch my RAW files. In addition, I would most certainly never do something dumb enough in PP to end up with my results, and then be all surpised as to where it came from - I believe I have an in-depth understanding of what I'm doing (please check out my DeviantArt gallery if you doubt this).

I am more than happy to post two RAW images: a 0.5s ISO3200 dark frame, and a "normal" photograph, which illustrates a severe pink cast, regardless of exposure time.

Where do you guys recommend I post them to?

thanks,
Dawid

Hi Dawid,

It will be interesting to find what the tech guy does. Hopefully, when he's done it, he will give it a thorough testing. I wonder if he knows what he is looking for? I mean, why was the original loaner such crap, too?

I think I'd have gotten canon into the loop by now, if it was a new purchase last October.

Best wishes,

Ray

Dawid Loubser said:

Where do you guys recommend I post them to?

Click to expand...

Yousendit seems to be used frequently for this purpose, or you could post them temporarily to a webspace where you may have some room left on.

Oh, and 2 black frames (taken some 6 seconds apart) are even more useful for analysis. they can be averaged for a better blackframe quality, and they can be subtracted for a temporal difference.

Bart

Hello,

I have managed to post the RAW file (for the small image I posted first in this thread) here (8.8mb CR2) - it's an ISO3200, 0.5s exposure.

Unfortunately only the one for now - I have terribly slow bandwidth today (and South African bandwidth is terrible to begin with, nothing resembling actual broadband).

The hot pixels really are not too bad on this image (though it's be nice if they can be completely mapped out - they are annoying). This is a true, correctly made dark frame - body cap closed, eyepiece shutter closed, and done in a pitch black dark closet (just in case). Absolutely no light reaching the sensor.

The technician (head technician for Camera Tek, the official Canon Professional Service Centre here in SA) actually seems very knowledgeable - he confirmed a sensor replacement, and now he's going to start replacing processing and power boards. He has already spoken to Canon, and recommended a replacement if this doesn't completely solve the problem. So there is of course always hope.

The original hypothesis stands - this looks more and more like severe amp glow (or some other component emitting heat?) Maybe it really isn't the sensor... But as to why the hot pixels are the same before/after the sensor change, I can't figure it out.

Oh, and he is aware of the problems with their loan body, but apparently they have no other 1D loan body available - one has to understand we're a really really small market down here. However, for the time being, I'll see if I can get them to loan me a 5D rather - at least the image quality will be better.

I will keep this thread up to date (and post some more RAW images when I have time, but have Mercy on our little server please!)

P.S. When I take a normal ISO3200 shot, and underexpose by 3 stops, instead of a normal dark (but, of course, expectedly very noisy image) I get a dark image overlaid with this same pink glow from the right, even if shutter speed is 1/200s. Happens in both JPEG and RAW.

Dawid,

I dl your cr2 and put in Silkypix, the bright pixels disappeared, but I'm a littlle dubious on the dark frame. There's a definite graduation (left to right) and significantly more noise that my 1s f/8 exposure, lens cap on and eye piece shut.

For your ref i've put mine here.

Dawid Loubser said:

John, thank you for your input, but forgive me if I assume it's somewhat flawed:

[*]I specifically stated that no adjustments were made, yet you assume I pushed the exposure, or don't understand my RAW converter

Click to expand...

Some RAW converters automatically try to make an image look properly exposed.

[*]I am intimately familiar with the noise characteristics of this camera model
[*]This is a true blackframe; taken with body cap on, and eyepiece shutter closed

Click to expand...

No, this is clearly a faulty camera.[/QUOTE]

I didn't say it definitely wasn't. I was explaining the color. I implied it was possibly defective when I said that the dark/light areas were a problem.

Have you tried a darkframe subtraction? That's the CF for noise reduction of long exposures.

I suspect that the blackpoint is inconsistent across the frame. I've seen this before on someone's 30D. Certain parts of the image were blacker than black. In your case, most of the frame seems to be brighter than black.

I don't think it's amplifier glow. Amplifier glow is localized to a small semi-circle on the right edge near the bottom of the frame.

If your hot pixel is in the same spot, I suspect that Canon didn't really swap out the sensor.

Click to expand...

Dawid Loubser said:

Hello,

I have managed to post the RAW file (for the small image I posted first in this thread) here (8.8mb CR2) - it's an ISO3200, 0.5s exposure.

Click to expand...

Thanks. That makes it much easier to see what is wrong.

The RAW blackframe is supposed to be an average of 128 DN (digital numbers, or RAW levels). Yours is all over the place. The dark edge along the top is 117, the upper left corner below the dark band is 147, upper right is 185, lower left is 146, lower right is 209. Noise is much higher than it should be; the standard deviation of a 1DmkII blackframe should be 9.6. Yours is anywhere from about 24 to 34 depending on the area.

As I suggested in my previous post, black is all over the place. The noise is about 3.5x as strong as it should be, as well. Your camera is quite ill.

The original hypothesis stands - this looks more and more like severe amp glow (or some other component emitting heat?) Maybe it really isn't the sensor... But as to why the hot pixels are the same before/after the sensor change, I can't figure it out.

Click to expand...

If it indeed is amp glow, it would seem that the amp is always on, and always heating the sensor, as the pattern suggests a slow and constant heating, as opposed to the small semi-circle you usually get in short exposures. Does the battery drain faster now?

P.S. When I take a normal ISO3200 shot, and underexpose by 3 stops, instead of a normal dark (but, of course, expectedly very noisy image) I get a dark image overlaid with this same pink glow from the right, even if shutter speed is 1/200s. Happens in both JPEG and RAW.

Click to expand...

The pink-ish magenta is just the way that daylight white-balance renders homogenous capture amongst the CFA filter neighbors.

Thanks so far

Thank you all for the input - and at least everybody has consensus that my camera is indeed very ill. I dropped it off this morning, and they will be replacing circuit boards today, as well as maybe try a different sensor (again).

I agree, in an app like SilkyPix the hot pixels are removed - they are easy to remove in many apps (unfortunately, Apple Aperture is not one of them :-( However, there is still a limit as to what is acceptable and what is not.

I'll keep you posted

John Sheehy said:

Some RAW converters automatically try to make an image look properly exposed.

Click to expand...

Not mine - I know how to use mine Apple Aperture (though not perfect) does not try to do this.

John Sheehy said:

Have you tried a darkframe subtraction? That's the CF for noise reduction of long exposures.

Click to expand...

Yes, with dark-frame subtraction, the camera renders a beautifully smooth ISO3200 1s image. But that doesn't really help me for normal exposures

John Sheehy said:

If your hot pixel is in the same spot, I suspect that Canon didn't really swap out the sensor.

Click to expand...

This is also very suspicious to me, but they said they did. Anyway, they said they'll try another sensor just for good measure (if swapping out the boards don't make a difference), so we'll see...

HI Dawid,

Dawid Loubser said:

This is also very suspicious to me, but they said they did. Anyway, they said they'll try another sensor just for good measure (if swapping out the boards don't make a difference), so we'll see...

Click to expand...

Sounds to Me that if they really did swap out the sensor & if that was the problem, they did not do the thermal calibration required.

A good friend of mine "Bart Pembroke" had strange problems with his 1Ds, at low temp 40 deg F or so his camera would shut off. After Canon Irvine (USA) did some sort of thermal calibration not only did the camera work OK below freezing, it also had much lower hi ISO noise.

Finally Solved! (with Details)

Hello all,

I am so happy to report that my serious problem has finally been resolved. In the end, the culprit was the circuit board "D". In addition, my camera has been fully calibrated and adjusted, and I must confess that the results are unreal. It's never looked this good. (I now finally feel I'm getting what I'm paying for - i.e. compared to, say, a 350D)

Under the same circumstances as my previous test images posted in this thread, here is a ISO3200 dark frame (this time I didn't bother with the dark cupboard) of 4 seconds, with noise reduction turned off (i.e. no dark-frame subtraction form the camera). I believe the results speak for themselves (I'm happy to post a RAW if anybody needs it). I have to push it to 10 seconds (no NR) to get visible hot pixels! I am a very happy customer...

Full resolution, JPEG compressed: http://ic3.deviantart.com/fs13/f/2007/093/1/3/Fixed_Sensor___For_Real_by_philosomatographer.jpg

Thank you all for the assistance.

Hi Dawid,

I'm glad its been fixed. Thanks for updating us. Now you can get away from pixel peeping and post some images with some other colours and patterns. I looked at the last raw you linked too, a week or so ago and as you were saying, it had problems - similar to the first. I suspect the hot pixels now - when you push it - are in far different places. I guess they did not change the sensor again.

This could be useful information for other folk who think they have sensor problems.

Best wishes,

Ray

You won't believe this...

...but, after a few days of blissful shooting with my camera, it developed yet another strange imaging problem. This time, regular tv-like lines through the image. Same intensity, regardless of ISO or exposure time (though cleaner images at low ISO actually show it better).

Having finally conceded that my camera is very ill, after all the time they've spent on maintenance and fault-finding, Canon South Africa has at least replaced my camera. I have a brand new black box to go and open tonight

Have a look at this scaled image, which includes a 100% crop:


Oh well, lightning doesn't strike twice... does it? I've never seen anything like this in all my experience with Canon DSLRs, so it must've been a freak occurrence. I post this here for academic interest - has anybody seen lines like these before? (though they are no longer my problem).