Open Photography Forums  
HOME FORUMS NEWS FAQ SEARCH

Go Back   Open Photography Forums > Digital Camera Discussion > Imaging Technology: Theory, Alternatives, Practice and Advances.

Imaging Technology: Theory, Alternatives, Practice and Advances. This is a brand independent discussion of theory, process or device. Ignore this forum unless this matters to you!

Reply
 
Thread Tools Display Modes
  #1  
Old November 2nd, 2015, 01:01 PM
Doug Kerr Doug Kerr is offline
Senior Member
 
Join Date: May 2006
Location: Alamogordo, New Mexico, USA
Posts: 8,418
Default Value compression of raw data in the Leica M8

It is a usual article of faith that the digitized outputs of the various sensor channels are linear with respect to the photometric exposure on the sensor. The proportionality will depend on the ISO setting (which affects the analog gain from the actual photodetector outputs to the analog-to-digital converters) and in general there is an "offset" (the digital output for "dark" is often non-zero).

And to date my examination of the details of this had been focused on the Canon EOS (and G-series) arrangements.

But my study of the detailed description of typical raw data processing in the DNG specification introduced me to a new possibility: that the digitized sensor output data might be "compressed" (in the value sense) before being delivered in a "raw" file. The advantage (just as when we compress the r, g, and b coordinates of an RGB representation into the nonlinear R, G, and B form that is actually "stated" for an RGB color space) is that fewer bits can be used per "sample" while still maintaining the same overall "perceptual" resolution (again in the value sense).

In such a case, a table (held in the raw file) is used during raw processing to transform the "compressed" raw data back to "linear" form for further attention.

I found that a raw file from a Leica M8 (which is in fact in the DNG form to begin with) in fact held such a table. I extracted it with Exiftool and examined it. (We note that apparently the delivered raw - DNG - file for this camera has the raw data in 8-bit precision.)

The table had 256 entries, presumably for delivered raw data values of 0-255. The outputs spanned a range just short of 0 to 2^14 (that is, can be considered to be to a precision of 14 bits).

The function of this "decompression" table is this:
y = x^2
where x is the delivered raw data value (on a scale of 0-1, scaled from its actual range of 0-255 for use in this equation) and y is the "decompressed" value (in this equation, on the basis of a range of 0-1, but in the actual table on a range of 0-16,383 - 14 bit precision).

This means that the "compression" done to the raw data before delivery is identical to a "gamma precompensation" function for gamma = 2.

By the way, for the Leica M8, the "black level" for the DNG file is stated at zero and the "maximum level" (often, but ill-advisedly, called "white level) is in fact stated as the maximum possible 14-bit value. Thus presumably the digitized data from the sensor has been "normalized" with regard to its range before applying the compression function. So for a "dark" photodetector, the delivered, compressed raw value would be 0 (8-bit representation). In processing, this would be converted by the table to the 14-bit value 0.

Very interesting.

Best regards,

Doug
Reply With Quote
  #2  
Old November 2nd, 2015, 02:43 PM
Asher Kelman Asher Kelman is online now
OPF Owner/Editor-in-Chief
 
Join Date: Apr 2006
Posts: 34,114
Default

Doug I thought the linear function was primarily the distinction of the format primarily used in video, otherwise there is a roll off approaching "black" and again "approaching white" in the form of an S curve with asymptotic extremes.

Asher
__________________
Follow us on Twitter at @opfweb

Our purpose is getting to an impressive photograph. So we encourage browsing and then feedback. Consider a link to your galleries annotated, C&C welcomed. Images posted within OPF are assumed to be for Comment & Critique, unless otherwise designated.
Reply With Quote
  #3  
Old November 2nd, 2015, 04:07 PM
Doug Kerr Doug Kerr is offline
Senior Member
 
Join Date: May 2006
Location: Alamogordo, New Mexico, USA
Posts: 8,418
Default

Hi, Asher,

Quote:
Originally Posted by Asher Kelman View Post
Doug I thought the linear function was primarily the distinction of the format primarily used in video, otherwise there is a roll off approaching "black" and again "approaching white" in the form of an S curve with asymptotic extremes.
I didn't think (in the case of still cameras) that happened at the sensor itself, but is rather applied in subsequent processing (typically subsequent to the delivery of the raw data from the camera).

The language of the DNG specification (which of course has an "indirect" relationship to these matters) seems to suggest that the behavior of the sensor itself (and the attending analog-to-digital converter) are expected to be at least nominally expected to be linear with photometric exposure ("linear light" is a common term for this).

Of course we know that in a typical digital still camera, there is a "tone curve" imposed as part of the processing of the sensor data into an RGB image (perhaps on the way to a JPEG file).

I had always assumed that one of the many advantages of taking the raw output from a camera is that we can modify such tone curves to suit our preferences for the particular shot.

But my knowledge of all this is very spotty and erratic.

Is it your thought that typically such an "S-curve" occurs by the time the sensor data is delivered in the raw file (when that is the form of output that is "taken"?

Thanks.

Best regards,

Doug
Reply With Quote
  #4  
Old November 2nd, 2015, 04:23 PM
Doug Kerr Doug Kerr is offline
Senior Member
 
Join Date: May 2006
Location: Alamogordo, New Mexico, USA
Posts: 8,418
Default

The above was one of several interesting things I learned about raw data handling in the Leica M8 camera.

An earlier one, brought to my attention by Asher (and I first I wasn't sure I believed there was any such thing!) was that in the Leica M8, correction for "vignetting" is applied in-camera to the raw data before it is delivered in a raw file (assuming that a "recognized" lens is aboard and that lens recognition is enabled).

The correction is defined separately for each of the camera's channels, thus allowing us to correct for "vignetting" that involves a chromaticity shift (as well as a luminance decline".

Because the mathematical definition of the radial variation in the luminance/chromaticity has several terms, we can by way of this "vignetting" correction process apply a chromaticity shift that is in fact uniform over the frame, thus allowing compensation for a "global color cast" imposed by a certain lens.

Again, all this is applied to the raw data before it is delivered from the camera.

I have not yet learned with certainty of other cameras in which this occurs.

Best regards,

Doug
Reply With Quote
  #5  
Old November 2nd, 2015, 06:42 PM
Asher Kelman Asher Kelman is online now
OPF Owner/Editor-in-Chief
 
Join Date: Apr 2006
Posts: 34,114
Default

Quote:
Originally Posted by Doug Kerr View Post
Hi, Asher,



I didn't think (in the case of still cameras) that happened at the sensor itself, but is rather applied in subsequent processing (typically subsequent to the delivery of the raw data from the camera).

The language of the DNG specification (which of course has an "indirect" relationship to these matters) seems to suggest that the behavior of the sensor itself (and the attending analog-to-digital converter) are expected to be at least nominally expected to be linear with photometric exposure ("linear light" is a common term for this).

Of course we know that in a typical digital still camera, there is a "tone curve" imposed as part of the processing of the sensor data into an RGB image (perhaps on the way to a JPEG file).

I had always assumed that one of the many advantages of taking the raw output from a camera is that we can modify such tone curves to suit our preferences for the particular shot.

But my knowledge of all this is very spotty and erratic.

Is it your thought that typically such an "S-curve" occurs by the time the sensor data is delivered in the raw file (when that is the form of output that is "
Yes, that's my current understanding. I need to find a reference to delineate my sense of it!

Asher
__________________
Follow us on Twitter at @opfweb

Our purpose is getting to an impressive photograph. So we encourage browsing and then feedback. Consider a link to your galleries annotated, C&C welcomed. Images posted within OPF are assumed to be for Comment & Critique, unless otherwise designated.
Reply With Quote
  #6  
Old November 2nd, 2015, 10:15 PM
Doug Kerr Doug Kerr is offline
Senior Member
 
Join Date: May 2006
Location: Alamogordo, New Mexico, USA
Posts: 8,418
Default

Hi, Asher,

Quote:
Originally Posted by Asher Kelman View Post
Yes, that's my current understanding. I need to find a reference to delineate my sense of it!
Yes, that would be helpful.

Thanks.

Best regards,

Doug
Reply With Quote
Reply

Bookmarks

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are Off
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
On the terms "lossless" and "lossy" compression Doug Kerr Imaging Technology: Theory, Alternatives, Practice and Advances. 25 September 21st, 2011 01:31 PM
Reading between the Lines: The Leica M9 Asher Kelman Breaking News 30 October 27th, 2009 05:24 AM
Leica returns the Whisper to M8! The new for ever upgrade policy! Asher Kelman Breaking News 4 February 6th, 2008 07:54 PM
Lossless compression and the Leica M8 scott kirkpatrick Layback Cafe 3 October 20th, 2006 07:52 AM


All times are GMT -7. The time now is 04:50 AM.


Posting images or text grants license to OPF, yet of such remain with its creator. Still, all assembled discussion 2006-2017 Asher Kelman (all rights reserved) Posts with new theme or unusual image might be moved/copied to a new thread!