• Please use real names.

    Greetings to all who have registered to OPF and those guests taking a look around. Please use real names. Registrations with fictitious names will not be processed. REAL NAMES ONLY will be processed

    Firstname Lastname

    Register

    We are a courteous and supportive community. No need to hide behind an alia. If you have a genuine need for privacy/secrecy then let me know!
  • Welcome to the new site. Here's a thread about the update where you can post your feedback, ask questions or spot those nasty bugs!

Measuring camera's saturation point in ADU; two different readings

Doug Kerr

Well-known member
Hi, Alex,

I don't have it, so I'd be grateful if you could send it to me.

Be glad to. Send me an e-mail so I will have your e-mail address:

doug.kerr@att.net

Very interesting. I suspected something like that. However, there still seems to be something off, as I did not get values within a certain range, but every single time only two distinct values.
Sure.

Hm, even more interesting. My values have never reached the max (and because of Canon's offset also not 0). Are you sure that this is the range of the actual RAW data and not the result of some scaling Rawnalyze performs with the white and black point setting (if it indeed does that; see below)?
I'm still figuring all that out.

I have never fully understood what the black and white point settings do in Rawnalyze (but then again, I didn't have a help file). Do those settings simply operate as thresholds of sorts and discard everything below and above them and then scale the data? Uncertainties like that made me move on to Iris...

Indeed!

Here is what the black point and white point settings do with regard to the pixels that are included in the statistics:

With Outside Range unchecked (to see this box you need to be in the Exposure display mode), then:

The sensels that are included in the statistics in the "black point" panel are those for which:

BP>=DN>2048 (more about 2048 in a minute)

where BP is the black point setting and DN is the data number (in ADU) of the sensel.

The sensels that are included in the statistics in the "white point" panel are those for which:

2017>DN>WP

where WP is the white point setting.

With Outside Range checked, then:

The sensels that are included in the statistics in the "black point" panel are those for which:

BP>=DN

The sensels that are included in the statistics in the "white point" panel are those for which:

DN>=WP

The significance of the 2048 is that values below that are considered to be in the "lower group" and values at or above that are considered to be in the "upper group". The mystic significance of this, or the premise for that number, has not yet some to me.

Now, as to the effect of those settings beyond the screening of value for the gathering of statistics, I am still sorting that out.

Couldn't the low values simply be noise at very dark tones that fluctuates around the offset?
I'm not sure exactly what you mean by "the low values".

Best regards,

Doug
 
That is very interesting. So you underexpose and then push it in post?

Hi Alex,

Yes, but only for 'high' ISO scenarios, high meaning probably a bit above unity gain ISO.
Here I described the results of several ISO settings to a Raw converted image. The underexposed images, e.g. dialing a minus EV correction in when in AV mode instead of upping the ISO, exhibit a lower standard deviation for noise.

Somewhat beyond the unity gain ISO, a higher ISO setting just amplifies the noise floor together with the signal at higher exposure levels, so no significant shadow detail is gained by using a higher ISO, but the camera's amplifier noise is added to the mix (before demosaicing). By pushing the exposure in the Raw converter, no noise is added.

As a bonus to lower noise, one also gains highlight protection which is welcome in low natural light scenarios where lighting is often contrasty and accurate measuring is difficult. Underexposing also allows to have a higher shutterspeed which can help in low light scenarios.

Cheers,
Bart
 

Doug Kerr

Well-known member
Hi, Alex,

Here is another interesting passage from the Rawnalyze help:

The tools of pixel selection are:

* the Black point and the White point values together determine the pixel value range, the black point giving the lower limit and the white point the upper limit of the range.

The current values are shown in the respective input field. The range of possible values for both limits is
o 0 to 16383 if the raw values are limited to 4095,
o 0 to 65535 if the raw values are limited to 16383,
o 0 to 131071 otherwise (this occurs with 16 bit depth, created by some digital backs).

The initial value of the black point is either
o the minimum of the black level correction values, if there are any; these are gained either from the descriptive data of the raw file, or calculated the so-called masked pixels around the image,
o 0 if black level correction does not apply.

The initial value of the white point is either
o the highest of the saturation/non-linearity levels of the pixels, if available,
o the white level specification from the descriptive data of the raw file,
o the maximum possible pixel value according to the bit depth of the sensor.​

I haven't completely figured out yet all its treasures.

Here is another:

Black level

Some cameras' raw images include several rows and/or columns of pixels outside the captured image as well; these are called masked pixels. Their values show the effect of the so-called black current, that part of the pixel value, which is not caused by the captured light.

Other cameras record one or more direct values indicating the black levels.

The pixel values of the image have to be corrected by the calculated values, depending on the pixel's position (typically with Canon cameras), or with the recorded correction value. Other cameras do not record any information for this purpose, rather they record the corrected pixel values.

The process described above is called the black level correction.

The initial value of Black point is the minimum of the black level correction values, if there are any, zero otherwise. However, this is only for orientation; when carrying out the black correction, the program always calculates with the pixel-related value, not with the minimum. [Huh?]

The minimum, maximum and average of the correction values are shown with the histogram display. [Labeled "black level".]

The first step of the conversion is substracting the respective black level correction value from the original pixel value. A negative result will be replaced by zero (yes, it can happen that the black level correction is higher than the pixel value).

In case the resulting value is zero, black clipping is said to have occured on raw level. "Clipping" in this sense is not a generally accepted term, but this is analogous to the clipping of highlights in the sense, that the value 0 represents all smaller values, which could not be recorded.

If the Raw clipping checkbox is checked, the black-clipped pixels will be displayed with the RGB value 255, or in case of composite color it contributes to the resulting color with 255. If the raw clipping option is not selected, such pixel values are converted to the RGB value 0, respectively they contribute to the composite color with 0.​

Further:

Black point

The result of the black level correction is an absolute pixel value, starting with 0. An adjustment can be applied to either increase the absolute values, or to clip them at a certain level; this adjustment is the black point setting.

If the black point is lower than the minimum of the black levels, then all absolute pixel values will be increased by the difference between the black point and the minimum of black levels. This causes the image to appear lighter; however, the relative differences between the pixels become less, i.,e. the contrast decreases.
Obviously, this can not happen if black level correction does not apply.

If the black point is higher than the minimum of the black levels, then all absolute pixel values will be reduced by the difference between the black point and the minimum of black levels. If black level correction does not apply, then the black point itself is the "clipping point".

This adjustment causes the image to appear darker, and details may be lost, as all pixel values between 0 and the black point will become "equalized", this is induced clipping, as opposed to raw clipping.

If the Show clipping checkbox is checked, then the pixels clipped by the black point but not clipped at the raw level will be displayed with the RGB value 255, or in case of composite color it contributes to the resulting color with 255. If the Show clipping option is not selected, such pixel values are converted to the RGB value 0, respectively they contribute to the composite color with 0.​

Another:

Pixel saturation

The pixels of some cameras can reach the maximum value representable with the given bit depth; for example the pixel values of the Canon 20D can reach 4095. Other cameras' pixels may clip at a lower level; the clipping level may depend on the ISO, on other factors, even on the particular copy of the camera model. It often depends on the channel (pixel color); even differences between the two green positions within the color filter array are not seldom. Some cameras' pixels (sometimes only certain channels) do not clip at a single level, but in a level range; this is the range of non-linearity. The pixels have to be regarded as clipped from the lower limit of this range, because the values within the range do not represent meaningful information about the image.

If the original (i.e. not black level corrected) pixel value is equal to or greater than the respective clipping level, then that pixel is said to be raw clipped. If the Raw clipping checkbox is checked, the clipped pixels will be displayed with the RGB value 0, or in case of composite color it contributes to the resulting color with 0. If the Raw clipping option is not selected, such pixel values are converted to the RGB value 255, respectively they contribute to the composite color with 255.

The current version of Rawnalyze contains the clipping levels of most supported cameras hard coded; however, those values may not be accurate with all copies of that particular camera model. This may lead to reporting raw clipping, when it did not occur, or not reporting actual clipping.​

And finally (for now):

White point

The white point specification, reduced by the current black point value sets an upper limit to the pixel values after black level correction and the application of the white balance and lightness adjustment factors. Thus the white point induces a clipping based on adjustments, as opposed to the raw clipping.

Reducing the specified white point by the black point is necessary, so that the white point value can be regarded relative to the uncorrected raw pixel values. Thus the white point gets "black point adjusted" in certain sense. If a pixel is not clipped on the raw level and the adjusted pixel value is equal to or higher than the white point and the Show clipping checkbox is checked, the clipped pixels will be displayed with the RGB value 0, or in case of composite color it contributes to the resulting color with 0. If the Show clipping option is not selected, such pixel values are converted to the RGB value 255, respectively they contribute to the composite color with 255.

Thus a pixel can show up as clipped only in response to one of the clipping indication options, not both.

The initial value of the white point is either

* the highest of the saturation/non-linearity levels of the pixels, if available,
* the white level specification from the descriptive data of the raw file,
* the maximum possible pixel value according to the bit depth of the sensor.

Decreasing the white point can induce clipping and it generally increases the contrast and lightness of the displayed image. Increasing the white point causes the image appear darker and less contrasty, but it may be necessary to avoid induced clipping, which can be caused by whute balancing and lightness adjustment. Turning on-off-on-off the clipping indication (by checking-unchecking the respective box) makes the affected pixels appear alternatingly very bright and very dark; this way it is easy to spot the affected areas.

Thus a pixel can show up as clipped only in response to one of the clipping indication options, not both.​

I have to go out and run some errands soon, but later today I will contemplate all this raw wisdom (!).

Best regards,

Doug
 

Doug Kerr

Well-known member
Hi, Alex,

It might be that all the numbers dealt with in Rawnalyze are on a scale whose origin is at the offset.

That is, if (for 14-bit machines) that offset is 2048, the numbers Rawnalyze deals with (and shows us) all have a range of 0-14335.

I have to read the stuff about how it determines the initial WP value to be able to see how that fits in. For example, on a 40D test file I just looked at - with the whole frame gravely blown out - the initial WP value is 13820, certainly not anomalous.

All the sensel values were reported as 13284.

The range chosen for the basic histogram is reported as 0-13849.

So my guess is that the offset is already taken care of as part of the definition of the number scale (which we never see happening).

This file was at ISO 100. For a similar file at ISO 400, the initial WP is 16384. The sensel values are about 16223 (there is some small variation - the exposure may not have been properly chosen for "full blowout"). The histogram range chosen is 0-16383.

But this is not consistent with the "working range" that would result from an offset of 2048 (0-14335). In fact it is precisely consistent with a zero offset (using the full available range). It is conceivable that the offset varies with ISO setting!

More after my errands!

Best regards,

Doug
 

Doug Kerr

Well-known member
Hi, Alex,

Is there any chance that the remarkable nearly 2048-unit discrepancy between the saturation levels in your two tests is a consequence of one of the test being done at one ISO setting and one at another, with the two bringing about different offset values (such as 2048 and 0)?

Just a thought.

Best regards,

Doug
 

Asher Kelman

OPF Owner/Editor-in-Chief
Hi Alex,

Yes, but only for 'high' ISO scenarios, high meaning probably a bit above unity gain ISO.
Here I described the results of several ISO settings to a Raw converted image. The underexposed images, e.g. dialing a minus EV correction in when in AV mode instead of upping the ISO, exhibit a lower standard deviation for noise.

Somewhat beyond the unity gain ISO, a higher ISO setting just amplifies the noise floor together with the signal at higher exposure levels, so no significant shadow detail is gained by using a higher ISO, but the camera's amplifier noise is added to the mix (before demosaicing). By pushing the exposure in the Raw converter, no noise is added.

As a bonus to lower noise, one also gains highlight protection which is welcome in low natural light scenarios where lighting is often contrasty and accurate measuring is difficult. Underexposing also allows to have a higher shutterspeed which can help in low light scenarios.

Cheers,
Bart

Bart and Alex,

Thanks for pionting out where these two phenomena work best. The reduction in noise at higher ISO for the same energy absorbed by the sensel set occurs at low ISOs whereas the benefit of taking the picture under-exposed at a lower ISO in low light occurs in the ~ 1600 ISO and above region.

I have been looking at DXO plots of measured dynamic range. The graphs can only hold 3 profiles. That for the Nikon D700 happens to follow closely to the Canon 5DII. As you can see the Fuji is remarkable until ISO 800. The Wonderful Nikon D3S tracks the D700 but higher by about one EV.

DXO_FujiS5pro.jpg

So here's the question. Can I use to advantage the giant plateau of super-dynamic range of the Fujfilm S5 Pro, ($2,000) up to ISO 800? Could I now could under-expose by even several stops and then easlly recover from RAW the highlghts and the shadows too.

At ISO 1600 I could use the Nikon D3S to do grab more headroom but at $5600.

Am I right n my interpretations and which is the best approach?

Asher
 
So here's the question. Can I use to advantage the giant plateau of super-dynamic range of the Fujfilm S5 Pro, ($2,000) up to ISO 800? Could I now could under-expose by even several stops and then easlly recover from RAW the highlghts and the shadows too

The dynamic range alone is not the whole story. When you look at the Signal to Noise ratio for the Fuji S5 Pro (click the Full SNR button) throughout the luminosity range, then the highest 4 stops towards saturation exhibit a much lower SN-ratio, i.e. the mid-tones to highlights are relatively noisy. It's a trade-off to get the extra DR.

The SN-ratios at ISO 800 for the top luminosity levels range from 23.4 -29.6 dB, compared to the Nikon D3s (click the Full SNR button) with 29.8 - 41.8 dB (6 dB is a factor 2). The D3s, although it has less shadow detail, has 2 - 4 x better noise performance in the upper part (half) of the range.

There are also differences in the color accuracy, with the D3s coming out on top, as should be expected for that money. Then there are the handling issues between the bodies, and the differences in Megapixel numbers and bit depth of the Raw data, which also translates to the smoothness of gradients.

They are hard to compare, but the Fuji image quality apparently does perform good with underexposure in the lower luminosity levels (which in turn reduces it's DR). With under-exposure of the D3s there is less lost in the shadows (because there is less to begin with) while image quality also stands boosting the exposure in postprocessing better.

Cheers,
Bart
 
Last edited:

Asher Kelman

OPF Owner/Editor-in-Chief
The dynamic range alone is not the whole story. When you look at the Signal to Noise ratio for the Fuji S5 Pro throughout the luminosity range, then the highest 4 stops towards saturation exhibit a much lower SN-ratio, i.e. the mid-tones to highlights are relatively noisy. It's a trade-off to get the extra DR.

The SN-ratios at ISO 800 for the top luminosity levels range from 23.4 -29.6 dB, compared to the Nikon D3s (click the Full SNR button) with 29.8 - 41.8 dB (6 dB is a factor 2). The D3s, although it has less shadow detail, has 2 - 4 x better noise performance in the upper part (half) of the range.

There are also differences in the color accuracy, with the D3s coming out on top, as should be expected for that money. Then there are the handling issues between the bodies, and the differences in Megapixel numbers and bit depth of the Raw data, which also translates to the smoothness of gradients.

They are hard to compare, but the Fuji image quality apparently does perform good with underexposure in the lower luminosity levels (which in turn reduces it's DR). With under-exposure of the D3s there is less lost in the shadows (because there is less to begin with) while image quality also stands boosting the exposure in postprocessing better.

Cheers,
Bart

Thanks for pointing out the other parameters of signal to nose ratio and color accuracy.

What you are saying then is that while it might work to my advantage at ISO 800 with the Fuji, it does not appear to be a good bet as far as noise s concerned! I was thinking of the Fuji since the price is easer to deal with. I am not concerned with pixel number as I can use a telephoto for individual performers and (then at the applause, use flash for the entire orchestra using the 5DII). So if a camera has a better ability to deal with low light, when flash cannot be used, keeping highlight detail and giving shadow detail in the eyes, then that's all I need.

Anyway, thanks for your help. I'm working much better since you advised me to use a one stop lower ISO and recover n RAW. I am wondering if there's a camera I can afford that do better!

Asher
 

Doug Kerr

Well-known member
Hi, Alex,

Well, very interesting results with some tests just done with the EOS 40D.

At each of five ISO settings (ISO 100, 200, 400, 800, and 1600) I took a "dark" frame (nominally unexposed) and a "well blown" frame (about 13 stops over the metered exposure). The files were examined in Rawnalyze.

The results were very consistent for ISO 200-1600 but different in one respect for ISO 100.

Dark frame DN values

In every case (for all ISO settings, ISO 100-1600), for the "dark" frame, the average DN was 1024 ADU (there were a few outliers above and below, more so for the higher ISO values). This certainly suggests that we are looking at the raw values without subtraction of the offset, and that the offset is 1024.

ISO 200-1600

Rawnalyze adopted a range for the histogram of 0-16384. (You might expect 0-16384, thinking of the range of 14-bit integers, but its scale is looked at as a number line 16384 units long (2^14), running from 0-16384, just as a 12" ruler runs from 0" to 12"). (We assume it is tipped to this my the camera profile.)

For the "well blown" frames, the average DN was in the range 16220-16223 ADU over the different ISO values (200-1600). This is about 99% of the available number space.

The initial black point was 975 for ISO 200, 950 for ISO 400-1600.

The initial white point was 16384 for ISO 200-1600.

ISO 100

Rawnalyze adopted a range for the histogram of 0-13850. (We assume it is tipped to this my the camera profile.)

For the "well blown" frames, the average DN was 13824 ADU. This is about 99.8% of the apparenty-allocated number space.

The initial black point was 944.

The initial white point was 13850.

I do not know why only about 84.5% of the theoretical 14-bit number space is used in the ISO 100 mode. This may fit in with something somebody else here knows.


Best regards,

Doug
 

Alex Weiss

New member
Hi all,

sorry for the delay.

I'm not sure exactly what you mean by "the low values".

Oh sorry. I was referring to this number:
In the one file I am looking at, there are values as low as 821 on the scale in use

Also, thank you very much for the excerpts from the Rawnalyze help file. Certainly a wealth of information. While I understand most settings now (I think so, at least), I'm still a bit uneasy about Rawnalyze messing with the data (or me being ignorant about certain settings), so I moved on to iris. Honestly, give it a try, the data there is about as raw as it gets ;) For further analysis, you can export a .fit file and analyze that further with ImageJ. I personally prefer that workflow.

Bart_van_der_Wolf said:
Yes, but only for 'high' ISO scenarios, high meaning probably a bit above unity gain ISO.
Here I described the results of several ISO settings to a Raw converted image. The underexposed images, e.g. dialing a minus EV correction in when in AV mode instead of upping the ISO, exhibit a lower standard deviation for noise.
Ah, that makes sense. Thanks for the link, Bart. Now I'm curious how my 550D would perform... Need to try that out :)

Doug Kerr said:
Is there any chance that the remarkable nearly 2048-unit discrepancy between the saturation levels in your two tests is a consequence of one of the test being done at one ISO setting and one at another, with the two bringing about different offset values (such as 2048 and 0)?
No, all of these tests were done at ISO100. I used different shutter speeds and apertures, though, but I don't think that should make a difference in terms of offset.
However, even though it is not applicable in my case, the thought that offset might be dependent on ISO is very interesting (and would make sense!). Another item on my list of things to check out :)

I do not know why only about 84.5% of the theoretical 14-bit number space is used in the ISO 100 mode. This may fit in with something somebody else here knows.
That is quite puzzling. Add to that the fact that I get two different ranges for the same ISO (ISO100), and we're facing a real mystery!
Again, thanks for measurements. I'll replicate your test and will check whether I get the same results.
 

Asher Kelman

OPF Owner/Editor-in-Chief
Any Effects of temperature on Saturation point in ADU?

Alex, Doug and Bart,

When shooting HD in the latest Canon DSLR's 12 minutes is all one gets, apparently, as the sensor becomes hot. So, is there a possibility that what you are measuring would be altered in intense still shooting as the sensor became hotter? Or because, perhaps, if you always use black frames in your measurements, the effect of possible temperature-induced noise is already corrected for?

Asher
 
Alex, Doug and Bart,

When shooting HD in the latest Canon DSLR's 12 minutes is all one gets, apparently, as the sensor becomes hot. So, is there a possibility that what you are measuring would be altered in intense still shooting as the sensor became hotter? Or because, perhaps, if you always use black frames in your measurements, the effect of possible temperature-induced noise is already corrected for?

Hi Asher,

I'm not all that familiar with the video functionality but I don't think the limit is primarily due to heat build-up. I think file size limitation is a factor, due to the recording medium's OS data structure (FAT). But I could be wrong in my assumptions, I have not tested it. I know there is a field in the EXIF of my 1Ds3 that records temperature, but I don't know where it is measured (maybe it's only used for the dust shaker functionality which does produce heat, and cannot be reused until after a certain timeout period).

One of the benefits of CMOS devices is that they don't use as much power as CCDs, and that's why they don't get as hot. They do build up some temperature, but it usually takes relatively long exposure times per frame before one can see it (usually as 'amp-glow').

- Black frames are used to quantify Read-out noise. They are taken at the shortest possible exposure time, in order to minimize heat induced noise. Most of the noise in those black frames is from the read-out process.
- Dark frames can be produced for Darkframe subtraction, these are taken at the same exposure time as the actual light exposures, but without exposing them to light. They differ from the blackframes only by exposure time, so their difference is attributed to temporal noise build-up. The noise becomes noticeable only at longer (roughly 1 sec.) exposure times.

I don't think the Saturation point is changed as a function of temperature. I think it is a factory/service setting based on the individual sensor array, but usually generic values are used per model/channel. Perhaps that is why the total range is not used for encoding, or for headroom for certain calculations (or signals for hot pixel mapping, just speculating). The maximum ADUs/DNs are just a quantization setting for the ADC, while the offset is to allow the recording of an unclipped base noise distribution which widens with ISO.

Cheers,
Bart
 

Jeremy Waller

New member
Book

Hi Chaps,

Here is a book that will be of interest.

CMOS/CCD SENSORS and CAMERA SYSTEMS

Gerald C. Holst & Terrence S. Lomheim

SPIE press



1. This is a really understandable book.

2. Sampling theory - very simplified - good to understand stuff.

3. Linear theory (Filters, MTF etc.) - very much simplified - very good to understand.

4. ADC spurious responses applied to images - well explained.

Regards,

Jeremy
 
Top