Open Photography Forums

Open Photography Forums (http://www.openphotographyforums.com/forums/index.php)
-   Image Processing and Workflow (http://www.openphotographyforums.com/forums/forumdisplay.php?f=38)
-   -   Lossy compression DNG files (http://www.openphotographyforums.com/forums/showthread.php?t=19677)

Doug Kerr August 4th, 2015 08:25 PM

Lossy compression DNG files
 
I have just become aware that there is such a thing as a lossy compression DNG file, which is smaller than the regular DNG file.

Using the Adobe DNG Converter on a CR2 file output by my Canon Powershot G16, where the CR2 file has a size of 14.6 MB, the regular DNG file has a size of 14.1 MB, and the lossy compression DNG file a size of 7.1 MB.

I have no idea how much of a compromise in the quality of the ultimately developed image might be encountered with the use of the lossy compression DNG file (or how we could even know).

The choice to use the lossy compression format for DNG files is found in the Preferences panel of the Adobe DNG Converter (accessible by pressing the Change Preferences button on the main page).

Best regards,

Doug

Tom dinning August 5th, 2015 12:15 AM

Quote:

Originally Posted by Doug Kerr (Post 163980)
I have just become aware that there is such a thing as a lossy compression DNG file, which is smaller than the regular DNG file.

Using the Adobe DNG Converter on a CR2 file output by my Canon Powershot G16, where the CR2 file has a size of 14.6 MB, the regular DNG file has a size of 14.1 MB, and the lossy compression DNG file a size of 7.1 MB.

I have no idea how much of a compromise in the quality of the ultimately developed image might be encountered with the use of the lossy compression DNG file (or how we could even know).

The choice to use the lossy compression format for DNG files is found in the Preferences panel of the Adobe DNG Converter (accessible by pressing the Change Preferences button on the main page).

Best regards,

Doug

Another quandary. I have the same suspicions when the snotty nosed, pimple faced teenager delivers the pizza. Did he add something and will it ultimately alter my experience?
The choice is simple: I don't buy pizza, nor do I use dng converter. It seems to be another step in the chain that could be tampered with.
Or am I just paranoid? After all, the said teenager is an ex student and I might have unfairly judged him.

Asher Kelman August 5th, 2015 02:27 AM

Quote:

Originally Posted by Doug Kerr (Post 163980)
I have just become aware that there is such a thing as a lossy compression DNG file, which is smaller than the regular DNG file.

Using the Adobe DNG Converter on a CR2 file output by my Canon Powershot G16, where the CR2 file has a size of 14.6 MB, the regular DNG file has a size of 14.1 MB, and the lossy compression DNG file a size of 7.1 MB.

I have no idea how much of a compromise in the quality of the ultimately developed image might be encountered with the use of the lossy compression DNG file (or how we could even know).

The choice to use the lossy compression format for DNG files is found in the Preferences panel of the Adobe DNG Converter (accessible by pressing the Change Preferences button on the main page).

Best regards,

Doug

Doug,

One way to look at this is to look at a picture of your good wife in a draw hat! If you process the RAW file with a set of Changes made in a separate layer in Photoshop, the correction layer is always copied from another file). With say 6 aggressive filters, if you look at the expanded histogram, do you see any jagged, "toothcomb", effect or on the image loss of detail in the edges of the straw or posterization near highlights or shadows especially??

If the histogram is smooth and your pictures seem flawless, likely as not, the compression in the DNG did no harm. Printing out the image would help too. Do sure Bart has more scientific tests!

Asher

Andrew Stannard August 5th, 2015 04:36 AM

Hi,

There is an article here that expands more on the lossy DNG Format:

http://www.pointsinfocus.com/learnin...-on-lossy-dng/

Alas I couldn't find an article to help with the quandary of Tom's pizza delivery...

Regards,
Andrew.

Doug Kerr August 5th, 2015 08:29 AM

Hi, Andrew,

Quote:

Originally Posted by Andrew Stannard (Post 163990)
Hi,

There is an article here that expands more on the lossy DNG Format:

http://www.pointsinfocus.com/learnin...-on-lossy-dng/

Thanks so much for that reference, which is very enlightening.

Quote:

Alas I couldn't find an article to help with the quandary of Tom's pizza delivery...
Well, there are many things still beyond our science!

Thanks again.

Best regards,

Doug

Doug Kerr August 5th, 2015 08:36 AM

Hi, Asher,

Quote:

Originally Posted by Asher Kelman (Post 163987)
Doug,

One way to look at this is to look at a picture of your good wife in a draw hat! If you process the RAW file with a set of Changes made in a separate layer in Photoshop, the correction layer is always copied from another file). With say 6 aggressive filters, if you look at the expanded histogram, do you see any jagged, "toothcomb", effect or on the image loss of detail in the edges of the straw or posterization near highlights or shadows especially??

I'm not sure I understand the part I have highlighted in red. Do you mean that whether I have the regular DNG file or the lossy-compressed DNG file loaded, I apply a consistent correction layer?

I don't know how to move correction layers from one file to another.

Interesting approach. Thanks for suggesting it.

Best regards,

Doug

Andrew Stannard August 5th, 2015 11:50 AM

Doug - Glad you found the article useful. I don't know if you followed the link in that page through to an adobe forum thread, but if you didn't it's worth doing so. It contains even further detail around the mechanism of compression used, and reasons why.

Andrew.

Doug Kerr August 5th, 2015 12:47 PM

Hi, Andrew,

Quote:

Originally Posted by Andrew Stannard (Post 164000)
Doug - Glad you found the article useful. I don't know if you followed the link in that page through to an adobe forum thread, but if you didn't it's worth doing so. It contains even further detail around the mechanism of compression used, and reasons why.

Yes, I did read the referenced thread. That is all way amazing!

Thanks again.

Best regards,

Doug

Doug Kerr August 5th, 2015 04:13 PM

Hi, Asher,

Quote:

Originally Posted by Asher Kelman (Post 163987)
One way to look at this is to look at a picture of your good wife in a draw hat! If you process the RAW file with a set of Changes made in a separate layer in Photoshop, the correction layer is always copied from another file). With say 6 aggressive filters, if you look at the expanded histogram, do you see any jagged, "toothcomb", effect or on the image loss of detail in the edges of the straw or posterization near highlights or shadows especially??

If the histogram is smooth and your pictures seem flawless, likely as not, the compression in the DNG did no harm. Printing out the image would help too. Do sure Bart has more scientific tests!

I don't understand the premise behind this suggested route of investigation.

Let's start by assuming that the CR2 file out of the camera will, when developed, lead to the desired image.

Next, let's assume that a normal DNG file generated from the CR2 file will do likewise.

Now we address a "lossy compression" DNG file generated from that same CR2 file. At issue, I believe, is to what extent does the use of lossy compression lead to this raw file developing into into a different image, perhaps one that does not as accurately reflect the scene. Maybe some detail is compromised in some way, or some artifacts emerge.

Is it your premise that if the lossy compression DNG file develops into a significantly different image than the normal DNG file, we can note that by subjecting the images developed from both DNG files to the same regimen of "aggressive" filter actions; if the file from the lossy compression DNG file is significantly different from the image from the regular DNG file, that will manifest by increased misbehavior under these aggressive filter actions?

Remember, the filter actions work on the image developed from the raw file by ACR. I can't quite grasp what unwanted differences in the image developed from the lossy compression DNG file might be expected to make that image more susceptible to misbehavior under aggressive filter actions.

What am I missing here?

Best regards,

Doug

Asher Kelman August 6th, 2015 02:05 AM

Quote:

Originally Posted by Doug Kerr (Post 163995)
Hi, Asher,



I'm not sure I understand the part I have highlighted in red. Do you mean that whether I have the regular DNG file or the lossy-compressed DNG file loaded, I apply a consistent correction layer?

I don't know how to move correction layers from one file to another.

Interesting approach. Thanks for suggesting it.

Best regards,

Doug


Hi Doug,

Add the correction layer to the file base layer with one version of the original file and then stage that layer to the image from the other, differently compressed file.

The object of the harsh testing is to discover whether or not the file still maintains great edges and gradations. Or perhaps the colors become posterized with sections of different hue, no longer gradually changing scrim one hue to another and instead jumping from A to B with obvious differences as in a comic book.

Of course one could use Imatest or some other program.

This is essentially the same question that jpg afficionados challenge RAW devotees with!

Asher

Doug Kerr August 6th, 2015 06:55 AM

Hi, Asher,

Quote:

Originally Posted by Asher Kelman (Post 164023)

Add the correction layer to the file base layer with one version of the original file and then stage that layer to the image from the other, differently compressed file.

I don't know how to do that with Photoshop. (Recall my skill with Photoshop is very limited!)

Quote:

This is essentially the same question that jpg afficionados challenge RAW devotees with!
Ah, yes.

Thanks.

Best regards,

Doug

Doug Kerr August 6th, 2015 10:00 AM

Here we see a visual comparison of a small crop from the JPG files developed (by Canon DPP) from two DNG files, generated by Adobe DNG Converter from same CR2 file generated by my Canon PowerShot G16 camera. The shot was at ISO 400.

The original shot resulted in a JPG image with dimensions of 4000 px 3000 px. The reconstructed JPG images (from the CR2->DNG chain) had those same dimensions.

The crop on the left is developed from a DNG file with normal compression; the one on the right from a DNG file with lossy compression. They are both presented on a pixel-for-pixel basis.

http://dougkerr.net/images/test/G03461A-C1.jpg http://dougkerr.net/images/test/G03461B-C1.jpg

Left: from DNG with normal compression. Right: from DNG with lossy compression.

Both developments were with the default settings of DPP.

We note that various clever steps are taken during the generation of a DNG file with lossy compression which may in fact have a beneficial effect on the noise in the ultimately recovered image.

Best regards,

Doug

Doug Kerr August 6th, 2015 12:02 PM

Well, something has obviously gone wrong in the preparation of my report above. Of course the two DNG files could not have been developed in DPP - it doesn't work on DNG files.

So I need to reconstruct exactly what I did do. I'll be back when I get that straightened out.

Sorry for the bum steer.

Best regards,

Doug

Doug Kerr August 6th, 2015 12:18 PM

Well, here is the report afresh, hopefully correct this time. The change is highlighted in blue (but the images are new also)

************

Here we see a visual comparison of a small crop from the JPG files developed (by Photoshop/ACR) from two DNG files, generated by Adobe DNG Converter from same CR2 file generated by my Canon PowerShot G16 camera. The shot was at ISO 400.

The original shot resulted in a JPG image with dimensions of 4000 px 3000 px. The reconstructed JPG images (from the CR2->DNG chain) had those same dimensions.

The crop on the left is developed from a DNG file with normal compression; the one on the right from a DNG file with lossy compression. They are both presented on a pixel-for-pixel basis.

http://dougkerr.net/images/test/G03461A-PS-C1.jpg http://dougkerr.net/images/test/G03461B-PS-C1.jpg

Left: from DNG with normal compression. Right: from DNG with lossy compression.

Both developments were with the default settings of PS/ACR.

I think I would have to say that I find a very slight deterioration in image quality overall from the lossy compression DNG route.

Best regards,

Doug

Doug Kerr August 6th, 2015 12:29 PM

Now, for something a little different, a similar comparison between:

(On the left) A crop from the JPG image developed by Photoshop/ACR from the regular DNG file generated from the camera CR2 file.

(On the right) A crop from the JPG image developed by Raw Therapee, using the vaunted AMaZE demosiaicing algorithm, from the regular DNG file generated from the camera CR2 file.


The noise performance is clearly inferior in the Raw Therapee development.

In addition, in the blue sky over the target house (no crops of that shown here) the noise in the RawTherapee development is greatly worse.

Of course, I did not optimize the various settings in RawTherapee (the development was with the default settings).

Interesting.

Best regards,

Doug

Doug Kerr August 6th, 2015 01:16 PM

Well, indeed I did not have noise reduction turned on in Raw Therapee.

Here are the two files again with noise reduction set to a modeate level in Raw Therapee:

(On the left) A crop from the JPG image developed by Photoshop/ACR from the regular DNG file generated from the camera CR2 file.

(On the right) A crop from the JPG image developed by Raw Therapee, using the vaunted AMaZE demosiaicing algorithm, from the regular DNG file generated from the camera CR2 file.


We see that this level of noise reduction seriously impacts image sharpness (note for example the license plate).

Of course again my settings may not have been "careful".

Just playing.

Best regards,

Doug

Doug Kerr August 6th, 2015 03:41 PM

Hi, Asher,

I said:
Quote:

Originally Posted by Doug Kerr (Post 163995)
Hi, Asher,

I don't know how to move correction layers from one file to another.

I figgered it out (at least two ways to do it actually).

Best regards,

Doug


All times are GMT -7. The time now is 04:09 AM.


Posting images or text grants license to OPF, yet of such remain with its creator. Still, all assembled discussion 2006-2017 Asher Kelman (all rights reserved) Posts with new theme or unusual image might be moved/copied to a new thread!