Open Photography Forums  
HOME FORUMS NEWS FAQ SEARCH

Go Back   Open Photography Forums > Digital Darkroom > Image Processing and Workflow

Image Processing and Workflow RAW, DNG , TIFF and JPG. From Capture to Ready for Publish/Display. All software and techniques used within an image workflow, (except extensive retouching and repair or DAM).

Reply
 
Thread Tools Display Modes
  #1  
Old August 4th, 2015, 09:25 PM
Doug Kerr Doug Kerr is offline
Senior Member
 
Join Date: May 2006
Location: Alamogordo, New Mexico, USA
Posts: 8,197
Default Lossy compression DNG files

I have just become aware that there is such a thing as a lossy compression DNG file, which is smaller than the regular DNG file.

Using the Adobe DNG Converter on a CR2 file output by my Canon Powershot G16, where the CR2 file has a size of 14.6 MB, the regular DNG file has a size of 14.1 MB, and the lossy compression DNG file a size of 7.1 MB.

I have no idea how much of a compromise in the quality of the ultimately developed image might be encountered with the use of the lossy compression DNG file (or how we could even know).

The choice to use the lossy compression format for DNG files is found in the Preferences panel of the Adobe DNG Converter (accessible by pressing the Change Preferences button on the main page).

Best regards,

Doug
Reply With Quote
  #2  
Old August 5th, 2015, 01:15 AM
Tom dinning Tom dinning is offline
Banned
 
Join Date: Mar 2012
Location: Darwin NT Australia
Posts: 2,088
Default

Quote:
Originally Posted by Doug Kerr View Post
I have just become aware that there is such a thing as a lossy compression DNG file, which is smaller than the regular DNG file.

Using the Adobe DNG Converter on a CR2 file output by my Canon Powershot G16, where the CR2 file has a size of 14.6 MB, the regular DNG file has a size of 14.1 MB, and the lossy compression DNG file a size of 7.1 MB.

I have no idea how much of a compromise in the quality of the ultimately developed image might be encountered with the use of the lossy compression DNG file (or how we could even know).

The choice to use the lossy compression format for DNG files is found in the Preferences panel of the Adobe DNG Converter (accessible by pressing the Change Preferences button on the main page).

Best regards,

Doug
Another quandary. I have the same suspicions when the snotty nosed, pimple faced teenager delivers the pizza. Did he add something and will it ultimately alter my experience?
The choice is simple: I don't buy pizza, nor do I use dng converter. It seems to be another step in the chain that could be tampered with.
Or am I just paranoid? After all, the said teenager is an ex student and I might have unfairly judged him.
Reply With Quote
  #3  
Old August 5th, 2015, 03:27 AM
Asher Kelman Asher Kelman is online now
OPF Owner/Editor-in-Chief
 
Join Date: Apr 2006
Posts: 32,802
Default

Quote:
Originally Posted by Doug Kerr View Post
I have just become aware that there is such a thing as a lossy compression DNG file, which is smaller than the regular DNG file.

Using the Adobe DNG Converter on a CR2 file output by my Canon Powershot G16, where the CR2 file has a size of 14.6 MB, the regular DNG file has a size of 14.1 MB, and the lossy compression DNG file a size of 7.1 MB.

I have no idea how much of a compromise in the quality of the ultimately developed image might be encountered with the use of the lossy compression DNG file (or how we could even know).

The choice to use the lossy compression format for DNG files is found in the Preferences panel of the Adobe DNG Converter (accessible by pressing the Change Preferences button on the main page).

Best regards,

Doug
Doug,

One way to look at this is to look at a picture of your good wife in a draw hat! If you process the RAW file with a set of Changes made in a separate layer in Photoshop, the correction layer is always copied from another file). With say 6 aggressive filters, if you look at the expanded histogram, do you see any jagged, "toothcomb", effect or on the image loss of detail in the edges of the straw or posterization near highlights or shadows especially??

If the histogram is smooth and your pictures seem flawless, likely as not, the compression in the DNG did no harm. Printing out the image would help too. Do sure Bart has more scientific tests!

Asher
__________________
Follow us on Twitter at @opfweb

Our purpose is getting to an impressive photograph. So we encourage browsing and then feedback. Consider a link to your galleries annotated, C&C welcomed. Images posted within OPF are assumed to be for Comment & Critique, unless otherwise designated.
Reply With Quote
  #4  
Old August 5th, 2015, 05:36 AM
Andrew Stannard Andrew Stannard is offline
Member
 
Join Date: Jul 2006
Location: Lancashire, UK
Posts: 371
Default

Hi,

There is an article here that expands more on the lossy DNG Format:

http://www.pointsinfocus.com/learnin...-on-lossy-dng/

Alas I couldn't find an article to help with the quandary of Tom's pizza delivery...

Regards,
Andrew.
__________________
Webpage
Blog
Twitter
Facebook Page
Reply With Quote
  #5  
Old August 5th, 2015, 09:29 AM
Doug Kerr Doug Kerr is offline
Senior Member
 
Join Date: May 2006
Location: Alamogordo, New Mexico, USA
Posts: 8,197
Default

Hi, Andrew,

Quote:
Originally Posted by Andrew Stannard View Post
Hi,

There is an article here that expands more on the lossy DNG Format:

http://www.pointsinfocus.com/learnin...-on-lossy-dng/
Thanks so much for that reference, which is very enlightening.

Quote:
Alas I couldn't find an article to help with the quandary of Tom's pizza delivery...
Well, there are many things still beyond our science!

Thanks again.

Best regards,

Doug
Reply With Quote
  #6  
Old August 5th, 2015, 09:36 AM
Doug Kerr Doug Kerr is offline
Senior Member
 
Join Date: May 2006
Location: Alamogordo, New Mexico, USA
Posts: 8,197
Default

Hi, Asher,

Quote:
Originally Posted by Asher Kelman View Post
Doug,

One way to look at this is to look at a picture of your good wife in a draw hat! If you process the RAW file with a set of Changes made in a separate layer in Photoshop, the correction layer is always copied from another file). With say 6 aggressive filters, if you look at the expanded histogram, do you see any jagged, "toothcomb", effect or on the image loss of detail in the edges of the straw or posterization near highlights or shadows especially??
I'm not sure I understand the part I have highlighted in red. Do you mean that whether I have the regular DNG file or the lossy-compressed DNG file loaded, I apply a consistent correction layer?

I don't know how to move correction layers from one file to another.

Interesting approach. Thanks for suggesting it.

Best regards,

Doug
Reply With Quote
  #7  
Old August 5th, 2015, 12:50 PM
Andrew Stannard Andrew Stannard is offline
Member
 
Join Date: Jul 2006
Location: Lancashire, UK
Posts: 371
Default

Doug - Glad you found the article useful. I don't know if you followed the link in that page through to an adobe forum thread, but if you didn't it's worth doing so. It contains even further detail around the mechanism of compression used, and reasons why.

Andrew.
__________________
Webpage
Blog
Twitter
Facebook Page
Reply With Quote
  #8  
Old August 5th, 2015, 01:47 PM
Doug Kerr Doug Kerr is offline
Senior Member
 
Join Date: May 2006
Location: Alamogordo, New Mexico, USA
Posts: 8,197
Default

Hi, Andrew,

Quote:
Originally Posted by Andrew Stannard View Post
Doug - Glad you found the article useful. I don't know if you followed the link in that page through to an adobe forum thread, but if you didn't it's worth doing so. It contains even further detail around the mechanism of compression used, and reasons why.
Yes, I did read the referenced thread. That is all way amazing!

Thanks again.

Best regards,

Doug
Reply With Quote
  #9  
Old August 5th, 2015, 05:13 PM
Doug Kerr Doug Kerr is offline
Senior Member
 
Join Date: May 2006
Location: Alamogordo, New Mexico, USA
Posts: 8,197
Default

Hi, Asher,

Quote:
Originally Posted by Asher Kelman View Post
One way to look at this is to look at a picture of your good wife in a draw hat! If you process the RAW file with a set of Changes made in a separate layer in Photoshop, the correction layer is always copied from another file). With say 6 aggressive filters, if you look at the expanded histogram, do you see any jagged, "toothcomb", effect or on the image loss of detail in the edges of the straw or posterization near highlights or shadows especially??

If the histogram is smooth and your pictures seem flawless, likely as not, the compression in the DNG did no harm. Printing out the image would help too. Do sure Bart has more scientific tests!
I don't understand the premise behind this suggested route of investigation.

Let's start by assuming that the CR2 file out of the camera will, when developed, lead to the desired image.

Next, let's assume that a normal DNG file generated from the CR2 file will do likewise.

Now we address a "lossy compression" DNG file generated from that same CR2 file. At issue, I believe, is to what extent does the use of lossy compression lead to this raw file developing into into a different image, perhaps one that does not as accurately reflect the scene. Maybe some detail is compromised in some way, or some artifacts emerge.

Is it your premise that if the lossy compression DNG file develops into a significantly different image than the normal DNG file, we can note that by subjecting the images developed from both DNG files to the same regimen of "aggressive" filter actions; if the file from the lossy compression DNG file is significantly different from the image from the regular DNG file, that will manifest by increased misbehavior under these aggressive filter actions?

Remember, the filter actions work on the image developed from the raw file by ACR. I can't quite grasp what unwanted differences in the image developed from the lossy compression DNG file might be expected to make that image more susceptible to misbehavior under aggressive filter actions.

What am I missing here?

Best regards,

Doug
Reply With Quote
  #10  
Old August 6th, 2015, 03:05 AM
Asher Kelman Asher Kelman is online now
OPF Owner/Editor-in-Chief
 
Join Date: Apr 2006
Posts: 32,802
Default

Quote:
Originally Posted by Doug Kerr View Post
Hi, Asher,



I'm not sure I understand the part I have highlighted in red. Do you mean that whether I have the regular DNG file or the lossy-compressed DNG file loaded, I apply a consistent correction layer?

I don't know how to move correction layers from one file to another.

Interesting approach. Thanks for suggesting it.

Best regards,

Doug

Hi Doug,

Add the correction layer to the file base layer with one version of the original file and then stage that layer to the image from the other, differently compressed file.

The object of the harsh testing is to discover whether or not the file still maintains great edges and gradations. Or perhaps the colors become posterized with sections of different hue, no longer gradually changing scrim one hue to another and instead jumping from A to B with obvious differences as in a comic book.

Of course one could use Imatest or some other program.

This is essentially the same question that jpg afficionados challenge RAW devotees with!

Asher
__________________
Follow us on Twitter at @opfweb

Our purpose is getting to an impressive photograph. So we encourage browsing and then feedback. Consider a link to your galleries annotated, C&C welcomed. Images posted within OPF are assumed to be for Comment & Critique, unless otherwise designated.
Reply With Quote
  #11  
Old August 6th, 2015, 07:55 AM
Doug Kerr Doug Kerr is offline
Senior Member
 
Join Date: May 2006
Location: Alamogordo, New Mexico, USA
Posts: 8,197
Default

Hi, Asher,

Quote:
Originally Posted by Asher Kelman View Post

Add the correction layer to the file base layer with one version of the original file and then stage that layer to the image from the other, differently compressed file.
I don't know how to do that with Photoshop. (Recall my skill with Photoshop is very limited!)

Quote:
This is essentially the same question that jpg afficionados challenge RAW devotees with!
Ah, yes.

Thanks.

Best regards,

Doug
Reply With Quote
  #12  
Old August 6th, 2015, 11:00 AM
Doug Kerr Doug Kerr is offline
Senior Member
 
Join Date: May 2006
Location: Alamogordo, New Mexico, USA
Posts: 8,197
Default

Here we see a visual comparison of a small crop from the JPG files developed (by Canon DPP) from two DNG files, generated by Adobe DNG Converter from same CR2 file generated by my Canon PowerShot G16 camera. The shot was at ISO 400.

The original shot resulted in a JPG image with dimensions of 4000 px 3000 px. The reconstructed JPG images (from the CR2->DNG chain) had those same dimensions.

The crop on the left is developed from a DNG file with normal compression; the one on the right from a DNG file with lossy compression. They are both presented on a pixel-for-pixel basis.



Left: from DNG with normal compression. Right: from DNG with lossy compression.

Both developments were with the default settings of DPP.

We note that various clever steps are taken during the generation of a DNG file with lossy compression which may in fact have a beneficial effect on the noise in the ultimately recovered image.

Best regards,

Doug
Reply With Quote
  #13  
Old August 6th, 2015, 01:02 PM
Doug Kerr Doug Kerr is offline
Senior Member
 
Join Date: May 2006
Location: Alamogordo, New Mexico, USA
Posts: 8,197
Default

Well, something has obviously gone wrong in the preparation of my report above. Of course the two DNG files could not have been developed in DPP - it doesn't work on DNG files.

So I need to reconstruct exactly what I did do. I'll be back when I get that straightened out.

Sorry for the bum steer.

Best regards,

Doug
Reply With Quote
  #14  
Old August 6th, 2015, 01:18 PM
Doug Kerr Doug Kerr is offline
Senior Member
 
Join Date: May 2006
Location: Alamogordo, New Mexico, USA
Posts: 8,197
Default

Well, here is the report afresh, hopefully correct this time. The change is highlighted in blue (but the images are new also)

************

Here we see a visual comparison of a small crop from the JPG files developed (by Photoshop/ACR) from two DNG files, generated by Adobe DNG Converter from same CR2 file generated by my Canon PowerShot G16 camera. The shot was at ISO 400.

The original shot resulted in a JPG image with dimensions of 4000 px 3000 px. The reconstructed JPG images (from the CR2->DNG chain) had those same dimensions.

The crop on the left is developed from a DNG file with normal compression; the one on the right from a DNG file with lossy compression. They are both presented on a pixel-for-pixel basis.



Left: from DNG with normal compression. Right: from DNG with lossy compression.

Both developments were with the default settings of PS/ACR.

I think I would have to say that I find a very slight deterioration in image quality overall from the lossy compression DNG route.

Best regards,

Doug
Reply With Quote
  #15  
Old August 6th, 2015, 01:29 PM
Doug Kerr Doug Kerr is offline
Senior Member
 
Join Date: May 2006
Location: Alamogordo, New Mexico, USA
Posts: 8,197
Default

Now, for something a little different, a similar comparison between:

(On the left) A crop from the JPG image developed by Photoshop/ACR from the regular DNG file generated from the camera CR2 file.

(On the right) A crop from the JPG image developed by Raw Therapee, using the vaunted AMaZE demosiaicing algorithm, from the regular DNG file generated from the camera CR2 file.



Left: PS/ACR development. Right: RawTherapee development

The noise performance is clearly inferior in the Raw Therapee development.

In addition, in the blue sky over the target house (no crops of that shown here) the noise in the RawTherapee development is greatly worse.

Of course, I did not optimize the various settings in RawTherapee (the development was with the default settings).

Interesting.

Best regards,

Doug
Reply With Quote
  #16  
Old August 6th, 2015, 02:16 PM
Doug Kerr Doug Kerr is offline
Senior Member
 
Join Date: May 2006
Location: Alamogordo, New Mexico, USA
Posts: 8,197
Default

Well, indeed I did not have noise reduction turned on in Raw Therapee.

Here are the two files again with noise reduction set to a modeate level in Raw Therapee:

(On the left) A crop from the JPG image developed by Photoshop/ACR from the regular DNG file generated from the camera CR2 file.

(On the right) A crop from the JPG image developed by Raw Therapee, using the vaunted AMaZE demosiaicing algorithm, from the regular DNG file generated from the camera CR2 file.



Left: PS/ACR development. Right: Raw Therapee development

We see that this level of noise reduction seriously impacts image sharpness (note for example the license plate).

Of course again my settings may not have been "careful".

Just playing.

Best regards,

Doug
Reply With Quote
  #17  
Old August 6th, 2015, 04:41 PM
Doug Kerr Doug Kerr is offline
Senior Member
 
Join Date: May 2006
Location: Alamogordo, New Mexico, USA
Posts: 8,197
Default

Hi, Asher,

I said:
Quote:
Originally Posted by Doug Kerr View Post
Hi, Asher,

I don't know how to move correction layers from one file to another.
I figgered it out (at least two ways to do it actually).

Best regards,

Doug
Reply With Quote
Reply

Bookmarks

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are Off
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
LensChanger: get your lens right in files! Nicolas Genette Image Processing and Workflow 4 March 9th, 2015 01:41 PM
JPEG compression level setting Doug Kerr Imaging Technology: Theory, Alternatives, Practice and Advances. 2 June 2nd, 2014 01:09 PM
On the terms "lossless" and "lossy" compression Doug Kerr Imaging Technology: Theory, Alternatives, Practice and Advances. 25 September 21st, 2011 02:31 PM
Faster saving of large multi-layer Photoshop files Doug Kerr Image Processing and Workflow 1 May 17th, 2011 09:19 AM
NEF "lossy" compression is clever Emil Martinec Nikon Mount DSLR's 15 June 14th, 2008 01:42 PM


All times are GMT -7. The time now is 03:55 AM.


Posting images or text grants license to OPF, yet of such remain with its creator. Still, all assembled discussion 2006-2017 Asher Kelman (all rights reserved) Posts with new theme or unusual image might be moved/copied to a new thread!