Deleting DNG - Restoring Original NEF

Hi all,

today i thought about deleting all my DNG Files and restoring the original NEF Files! Stupid, isn't it?
LR1.0 is out and it is time to decide whether to go the DNG road or stay with the original NEF as the main storage format

My points are:

1) It does not make sense to keep both formats. So either the NEF or the DNG has to go. Embedded NEF is not a real option - imho it's just a waste of space.

2) DNG is no real standard. IIRC only Leica uses DNG as native Raw format. I think it is not more future proof than any other format.

3) LR, ACR and most other converter are supporting both formats. As far as i understand DNG is based on tiff but also on the format it was converted from. RawShooter was able to handle DNG files only if the original format was also supported. This looks like DNG is not really "device independent". If one day my current NEF is no longer supported, i think i will also get problems with my DNGs from these NEFs! So why should i convert my original file to dng and delete it?

4) Nikon Capture NX only supports NEF Files. I don't use it right now, but what if Nikon decides NX 2.0 or NX 3.0 will be freeware or bundled with my next D300 which doesn't have a bayer type sensor? Or the next generation of Nikon View will be the "perfect" organization an management tool for NEFs?

5) I do not see any advantage in developing a DNG instead of developing a NEF within LR or ACR.

6) Every time i convert my NEF i have the strange feeling i do something wrong. Isn't it silly to convert my original files to a different format with no real advantage (ok: maybe 10% smaller filesize) and then delete the originals???

So, please tell me: Why do *you* convert your rawfile to dng and delete it afterwards?? Isn't it only just two more steps in the workflow??

Thank you for your answers!

Best regards
Reinhard

Hi Reinhard,

I don't yet use DNG myself, but I am considering doing so as recommended by Peter Krogh in his DAM book. The only real reason for doing so IMO is due to the fact that keywords and IPTC data are embedded in a DNG file itself and not saved to a sidecar file (.xmp). Also, a DNG file can contain a colour corrected full sized preview of the RAW image, which can be viewed without having to render the RAW file.

I don't believe in the theory/motivation that recommends the use of DNG due to the possibility that propriatery formats may disappear someday leaving unreadable files behind. When and if we get to that point, we'll have some transition period during which one can batch process his/her old format files and convert them into an actual format, not very unsimilar to the process of migrating one's back-up data from older storage (floppy, zip drive, tape, CD-R, etc) to newer ones (external HD, DVD-R, HD-ROM, Blue-Ray, etc). When the moment comes, I'll choose the most useful new format for converting my Canon raw files into, but I'm not certain that it will be DNG. DNG itself might disappear too.

These are my conclusions so far but since I am not an expert on DNG, I might be missing some essential info. Therefore, I am very curious as to the authoritative answers/insights we'll hopefully get in this thread .

Cheers,

Reinhard,
I only used DNG at a time when I had Sony 828, whose raw files were HUGE and not supported by then-new ACR. Conversion to DNG made total sense.
Since that time I switched to Canon line and found in practice that win Canon RAWs DNG format has zero benefits, only hassles. Extra step=extra time needed, eats 100% more HDD space...
My current mode of using DNG converter is when I want to give somebody a raw file but don't want to give the actual original. In this case I simply convert to DNG (without including the original) and git that DNG. This kinda guarantees that my real digital negatives, canon CR2 raw files, always stay with me and nobody can claim the authorship but me.

HTH

Cem,

Cem Usakligil said:

....The only real reason for doing so IMO is due to the fact that keywords and IPTC data are embedded in a DNG file itself and not saved to a sidecar file (.xmp). Also, a DNG file can contain a colour corrected full sized preview of the RAW image, which can be viewed without having to render the RAW file. ...

Click to expand...

The reasons you mentioned are - for me, that is - even more the reasons NOT to use DNG.
You see, as a programmer I know from a bitter experience that once you open a file "for writing", there is always a non-zero chance that you'll end up with a "corrupted" file.

Having all the adjusted settings stored in the separate "sidecar" XMP file guarantees that my digital negative is as safe as it can possibly be, and unless the hdd itself bites the dust, nothing will happens to it.

FWIW...

Cheers!

Nikolai,
i'll second your opinion. The original raw file should be "read only" from the moment it is transfered to your HD! Then you have always an untouched original raw file. So far i didn't have had any fileproblems with my 10,000 dngs.

Cem,
storing IPTC within DNG is also no advantage, since this is only of interest when the picture is used by another person. In that case i will export a jpg of that pic and apply the IPTC to the jpg. My raw file can stay read only.
Storing keywords in the file is in fact an advantage when i change my picture managementsystem (LR right now). But then i will get even more problems, since i will loose also the data stored in the LR database (collections etc.).

Cheers
Reinhard

Hi Reinhard,

Personally, I'm a firm believer in DNG. I convert all my RAW files to DNG and I don't keep backups of the originals.

I know of others that convert to DNG for their main database, but keep original RAW backups elsewhere.

I think that the important point about using DNG or not is too fully understand the advantages and disadvantages of doing so. From there each person can make up his or her own mind about the path to follow. You seem to have a good grasp of the concepts, however before making your final decision, I'd recommend that you read the first chapter of the DAM book which discusses the fors and againsts of DNG. Luckily the first chapter is available as a free download from here:

http://www.thedambook.com/

Regards,

Tim

Reinhard Pfaff said:

So, please tell me: Why do *you* convert your rawfile to dng and delete it afterwards??

Click to expand...

I don't. IMO this is a significant problem with LR. It would be a *much* better tool if it incorporated real browser functionality and was able to catalog ANY file in its existing format. As it stands, it does too much to alter files behind the scenes for my liking.

Cheers,

I wouldn't recommend that anyone convert to DNG and then destroy their original proprietary RAWs. That's just too chancy a proposition for my liking.

Instead, I archive my original RAWs to CD with a thumnail print of the shots inserted in the CD slim jewel case cover. That allows me to flip quickly through my collection and find stuff, with or without having a computer-based cataloging system available.

Then I convert to DNG and store the original RAW in the DNG file, renaming and filing the created files in the process, using a date/place nameing scheme of file names/folders. I know that placing the original RAW in the DNG increases the file size, but disk space is CHEAP these days. That really shouldn't be a consideration. You can put together a network-attached storage server with 500GB of auto backed-up space for a couple of hundred bucks.

At that point, I can feel free to assign keywords, and do other kinds of manipulations on the DNG files, knowing that I can either extract the original RAW at some point and start over, if need be, or find the original RAW on my archive CDs and go from there.

The point of DNG is that it is documented, and Adobe has freely licensed its use for just about anyone that is interested in supporting the format, both in-camera and in processing/archiving/viewing programs. This, more than anything else, future-proofs the format.

Absolutely no other camera maker has ever publicly documented their proprietary RAW formats, and there are already cases of older formats being 'orphaned' by a camera maker, where their current software on current supported operating systems no longer will read their older formats. The OEM in question also did not provide any 'upgrade' path for those unfortunates. Their only choice has been to go to third-party software applications, like Photoshop, to process those old images. Unless you want to mothball your old computer/operating system, you haven't much choice in a situation like this. And remember, Microsoft has sunsetted their support for older OSes for security patches and updates, so you wouldn't dare expose an older machine/OS (like Windows 98) to the Internet. An unpatched machine these days lasts for about 30 minutes, tops, before its invaded by all the script kiddies out there...

Jack_Flesher said:

I don't. IMO this is a significant problem with LR. It would be a *much* better tool if it incorporated real browser functionality and was able to catalog ANY file in its existing format. As it stands, it does too much to alter files behind the scenes for my liking.
Cheers,

Click to expand...

Converting to DNG during import is an option. LR is quite capable of reading practically any RAW format that you should feed it.

My photo storage practices follow Dave New's very closely. Concerns about corruption on writing metadata are highly exaggerated in my opinion; the chance that a hard drive will fail is much higher, and in either case, I have several backups on different media to rely on.

Aside from integrated metadata storage, the most important reason that I convert to DNG with embedded NEF is to simplify my folder structure. Having everything in one file means that I can avoid a proliferation of folders under each "shoot" folder - or a proliferation of folder trees, if I were to keep different formats under different top-level folders.

Concerning metadata storage, I greatly prefer to have everything contained within the DNG. Having a bunch of XMP files is a pain whenever I need to do any file management from the operating system, and I don't have any good reason not to have the data integrated into the DNG. IPTC data is useful in Lightroom, as I can have photos sorted by location or some other property based on the data associated with each photo. This metadata is especially important in general because a shoot may contain subjects that aren't listed in the folder name, and all photos are renamed to a very generic template, e.g. from "DSC_1234.NEF" to "jliechty_20070301_1234.dng" upon conversion. With proper tagging, I can even search from the operating system and get reasonable results, albeit slowly and somewhat inconveniently.

Tim,

if i decide on keeping on converting my nef to dng, i will remove the nefs. The nefs on dvd as a backup is no good option. A backup has to mirror my working environment. Therefore i us microsofts synctoy for this job and mirror my files (pics and more) to an external disk. A nef on dvd is *no* backup for a dng in progress!

Jack,

browsing within LR is better than browsing in any other browser i know. For presenting my pics to someone else i export then to jpg.

Dave, Josh,

disk space is cheap. i fully agree. But modern Software will need it. Vista takes around 40 GBytes and more and more packages come with video documentation and tutorials. What about the new dslr around the corner: 20 Mpixel will result in a 40 MByte dng. Not bad! You will need your disk space!

Tim,

i will have a closer look on the options for importing and converting to dng within LR. Up to now i move the nefs from the card to my disk and convert/rename them in a second step.
BTW: My naming convention is very similar to Joshs. My folder structure is also date oriented: Year->Year_Date_ShortDescription.

No final decision made!

Cheers
Reinhard

Reinhard,

I must disagree with your viewpoint that having the original NEFs archived are of no use. I consider the original RAWs to be my first generation digital negative, and in spite of anything else that might go wrong, I can always go back to my original RAW shot and start over if need be, as long as I can get to my original RAWs. Converting to DNG and embedding the original RAW gives me my 2nd layer of insurance against the eventuality that I might not be able to access my original RAW files with current OEM software, or that my CD/DVD backup media of my original RAWs might fail.

If I toss my original RAWs, and keep only my latest 'work in progress', then if something goes wrong, I have no way to recover and start over.

I used to work for a tape backup company (world's largest manufacturer of mini-cartridge tape backup drives for IBM PCs, MACs, and Novell servers), so you can see where I get my paranoia from. I had heard many heartbreaking stories from customers that didn't pay attention to basic data safeguarding practices and lost a lot of unrecoverable work.

Even after I scan in all the negs/slides I care to from my previous film days, I still won't discard the originals. I guess I'm just that kind of packrat.

Dave,

the question is whether the converted dng is as good as the original nef! If this is true, then there is no reason to keep the nef *and* the dng. Nobody needs two originals. No two original text documents, no two original c++ files, no two original pictures.
Adobe Photoshop Lightroom (and many other raw converters) is a non-destructive picture processor. You can go back to the very first step anytime and start over if you like to do that.
I'm aware of the importance of a good backup strategy. Synchronizing my nef/dng together with the LR Database to an external harddisk matches my backup needs fully. The original nef/dng is covered and my "work in progress" also. An embedded nef within a dng is redundant! With LR i can start over with both formats: nef and dng.

Another question: Who has converted the original raw picture to dng and sometimes later started over with the original raw file (instead with the converted dng)? And if so: why didn't you start over with the dng?

cheers
Reinhard

Reinhard Pfaff said:

Dave,

the question is whether the converted dng is as good as the original nef!

Click to expand...

That's the point. The DNG converter tries to maintain all the proprietary information that the OEM may be hiding in their 'maker' area (and other, possible undocumented areas), but Photoshop and brethen sometimes cannot fathom the intent/purpose of some of those proprietary fields.

I admit that embedding the original RAW within the DNG should allow you to retrieve the original RAW in the future, in case you wish to use the OEM's proprietary RAW converter software on it, or if an improved DNG converter and/or 3rd-party RAW converter comes to light.

The danger is that *any* manipulation of the original RAW data, even if it is simply to imbed it in another file format, can produce a mangled data file, either through a program bug, or hardware failure in the media reading/writing process.

I still stand by my conviction that one or more copies of the original RAW files on read-only media is your best insurance policy against all kinds of bugs/failures hardware/software, or purposely destructive, as in a virus attack.

Another question: Who has converted the original raw picture to dng and sometimes later started over with the original raw file (instead with the converted dng)? And if so: why didn't you start over with the dng?

cheers
Reinhard

Click to expand...

I have, on several occassions. Mainly the reason being that once I've archived my original RAW file, I have a more relaxed attitude about ripping into my DNG formatted files. Sometimes, I've found it more convenient to start over again from scratch with a particular file, than to untangle the various potentially destructive edits I've made. This has happened more often when I'm experimenting with automated scripts for ingesting, keywording, noise reductions, etc.

Cheers,

-- DaveN

Dave New said:

The DNG converter tries to maintain all the proprietary information that the OEM may be hiding in their 'maker' area (and other, possible undocumented areas), but Photoshop and brethen sometimes cannot fathom the intent/purpose of some of those proprietary fields.

Click to expand...

Indeed! As a matter of fact, the Adobe Raw Converter initially misinterpreted / ignored part of the Canon Raw data that described the sensor elements beyond the periphery of the image data, the 'blackened' sensels that carry information about systematic read-noise.

I admit that embedding the original RAW within the DNG should allow you to retrieve the original RAW in the future, in case you wish to use the OEM's proprietary RAW converter software on it, or if an improved DNG converter and/or 3rd-party RAW converter comes to light.

Click to expand...

Agreed, but it would only add to the DNG file size, where the original Raw file is smaller.

The danger is that *any* manipulation of the original RAW data, even if it is simply to imbed it in another file format, can produce a mangled data file, either through a program bug, or hardware failure in the media reading/writing process.

Click to expand...

I agree, and I'm commonly not known to be paranoid about conspiration theories, bu these things can happen.

I still stand by my conviction that one or more copies of the original RAW files on read-only media is your best insurance policy against all kinds of bugs/failures hardware/software, or purposely destructive, as in a virus attack.

Click to expand...

Yes, it's blindingly simple: At least Archive The Original.
It probably takes less space than other non-lossy formats and it is a safeguard against errors that others/oneself might make.
Whether on multiple hard disks, or CD/DVD, one should at least store the foundation of ones work. Subsequent 'interpretations' will only reduce the chance of total loss.

I have, on several occassions. Mainly the reason being that once I've archived my original RAW file, I have a more relaxed attitude about ripping into my DNG formatted files. Sometimes, I've found it more convenient to start over again from scratch with a particular file, than to untangle the various potentially destructive edits I've made. This has happened more often when I'm experimenting with automated scripts for ingesting, keywording, noise reductions, etc.

Click to expand...

Storing the original will buy a lot of peace of mind. Any subsequent 'fiddling about' cannot hurt the original data. It might not be fully integrated in a DAM solution, yet, but there will be a fall-back scenario should/when things go awry.

As to the media to use for this 'mother of all originals', make sure you use a 'quality' drive/media combination. The results of CD/DVD drives on various media, well, varies. One should attempt to select the most compatible combination. My Plextor drives not only come accompanied by suggested optimal media, but they also include a utility to verify the quality of written data, and it allows to do so as time goes by.

It is not very useful for an application to verify that the medium can/cannot be read at a given moment in time, some advance warning is needed instead. The Plextools utility will allow, with appropriate recorders, to evaluate the (sofar) successful error correction levels, before things get beyond repair.

Bart

Pre-Lightroom I used Portfolio for my database and needed the color-adjusted previews of the DNGs for it to be useful (unadjusted Canon 1Ds previews look hideous). So I would burn a DVD of the original camera raw files and then have the DNG files on a live HDD. Now that lightroom creates its own previews, having the embedded previews in the DNGs is a non-issue. The problem with my workflow here is that I do not have a setup to backup the metadata changes that I make in these DNGs while working day to day. I have switched back to just using camera raw files because backing up changes to .xmp sidecar files seperately requires a lot less space and time than backing up a whole DNG, and I don't need to spend the time creating the DNGs in the first place.

Dave New said:

The danger is that *any* manipulation of the original RAW data, even if it is simply to imbed it in another file format, can produce a mangled data file, either through a program bug, or hardware failure in the media reading/writing process.

Click to expand...

This is absolutely true because I have had lightroom destroy both metadata (on thousands of photos) and image data (only on one image that I know about) during its many crashes. Fortunately, I just so happened to have backups of those metadata-embedded DNGs because I had just copied them from my external HDDs to the internal RAID in my new box and hadn't erased the external HDDs. So now I have one working set of files and two backups...

Is there any way to take metadata from DNGs and export it to .xmp sidecar files that can then be attached back to the original RAW files, if the raw files are not embedded in the DNGs?

Hi,

after reading all the answers, discussions, whitepapers and the first chapter of Peter Krogh's book i will stay with dng.

My Workflow will look like this:

1. Import from the CF Card using nikon's picture project. This allows me to rename the nef files to my naming scheme (PR-YYYY-MM-DD-###.nef). The folder is "...\OriginalArchive\YYYY\YYYY-MM-DD-Description". The files on the CF Cards are deleted!

2. A first selection is done within PictureProject.

3. "Copy, convert to dng and import" within LR. No changes to the filename or to the foldername are necessary. The folder and the filenames of the nef-files and the dng-files are the same. The original nef-file is *not* embedded in the nef. All further processing is done based on the dng file.

4. The original nef files are moved(!) to dvd as soon as a dvd gets filled.

5. My backup is done with synctoy (mode: echo) to an external harddisk. The nef files are included as long as they "live" on my primary harddisk. After moving them to a dvd, they are also removed from the backup (synctoy mode "echo" does this automatically).

I know, i can ingest the files also with LR. But LR does not rename the original nef files and the folderstructure of the backup is also different. The renaming of the imported files is very good, but the backupfiles are not changed. Also i have to delete the files on the CF Card in a separate step.

I think this strategy allows me to implement an efficient and fast workflow based on dng-pictures, while preserving my original nef-files.

Cheers
Reinhard

Actually sounds quite good, although I like the idea of leaving the original file name alone, and having DNG keep track of it within the DNG. Just a preference, I suppose.

The other thing is that I dislike having anything on the PC write or delete files on the flash card, so I delay deleting the original RAW files on the flash until I'm ready to re-use it in the camera. At that point, I do a 'format card' in-camera, which puts a fresh FAT table on the card. This encourages images to be written sequentially to the card, which hopefully would make it easier to use a recovery utility to retrieve munged files on the card, if necessary.

I place my 'ready-to-be-reused' cards face-down in my flash card case, so I can easily tell the ones that have had their images transferred to the PC from ones that may contain live, non-transferred images.

Dave,

deleting the files on the flash card after transfering them to the pc has allways been a matter of course. The pics on my cards are allways transfered to the pc asap. Never had any problem with my cards although i format them once a month (even if they are empty).

Reinhard

Am i paranoiac?

Hey,
I might be a bit on the paranoiac side ...

Workflow is as following:
1. shooting (raw=NEF from Nikon D200)
2. import into lightroom from CF-Card (and transform to DNG) and do selections with 'client'
3. after session copy content of CF-cards to macbook pro
4. burn 2 copies on DVD
5. copy to different external disks and machines after the session
6. format cf cards in camera (so that i am ready to go on the next session)
7. take 2 sets of dvd to office / friend's home
8. depending on volume copy lightroom (db with pictures) to 2 different locations
9. after major change of OS / new software / much new data the internal disk of macbook pro gets cloned
10. PS'd pictures get added back to lightroom so i keep the original DNG and the final PSD in lightroom

Main system is a macbook pro in the studio, P4 Windows XP PRo (3 GB RAM, 0.8TB disks) as editing machine.

All systems connected with a gigabit LAN.

I have a backup on 1 external disk which is constantly connected but not switched on.
Backup No2 is an external disk which gets connected once a week
Backup No3 is on a different machine Mac G5 internal 500GB disk
Backup No4 is on one of the editing machine's internal 500GB disk
Backup No5 is on an external disk to the editing machine.
Backup No 6 consists of 2 different sets on DVD: on in the office, the 2nd with a friend
Another external disk which has a 1to1 copy of the internal macbook pro disk.

all disks were purchased in a different time frame, from different brands

The volume is not yet too important: in 9 month about 170 GB or raw pictures (about 14000), lightroom about 60 GB.

The lightroom environment is backed up as well on 2 different locations.

Next plans are to replace the different external disks by at least a couple of NAS (1..2 TB each)

Yes i suffered data loss, but not with the current systems, and not at home.


Nicolas

Dave New said:

The other thing is that I dislike having anything on the PC write or delete files on the flash card, so I delay deleting the original RAW files on the flash until I'm ready to re-use it in the camera.

Click to expand...

Dave,

you are right. Currently i ran into problems with one of my cards. While transfering the files from the card to my pc, i got an error message whitch results in a loss of half of the pictures. Not a single file was visible, neither in PictureProject nor in the filesystem. Fortunately the rest of the pictures were accessible in the camera, so i was able to transfer them via cable to the pc!
I think, it will be better to access the cards in an read only mode!

Nicolas,

for me this wouldn't work. I have three storage units:
(1) dvds to backup the original NEFs,
(2) one primary harddisk for storing the NEFs, DNGs and TIFFs,
(3) one external hardisk to backup my primary harddisk (file based).

I know, this covers not all risks, but this strategy adresses most of the daily risks (hard disk crash, accidently deleting of files) and i'm able to start over with the original DNG or even with the NEF, if i need to do so.

Regards
Reinhard