Open Photography Forums  
HOME FORUMS NEWS FAQ SEARCH

Go Back   Open Photography Forums > Digital Darkroom > Digital Asset Management: access & storage of image files.

Reply
 
Thread Tools Display Modes
  #1  
Old June 24th, 2006, 12:00 PM
Eric Perlberg Eric Perlberg is offline
New Member
 
Join Date: Jun 2006
Posts: 7
Default Longer term storage

I'm just getting to the point where I need to develop a serious storage strategy rather than a kind of adhoc affair. At the moment I simply back up both RAW images and processed images onto 2 internal hard drives. When I get 250 gig or so of images, I buy a cheap usb2 hard drive, copy RAW and Processed files to the new drive, make sure they're really there, then pack it all up in its original packing with power supply and cable and put it in the closet. Then I clear the data from the internal drives. I could easily convince myself to back this data up twice because drives fail. But I can see that as a slippery slope leading me to remortgaging, etc. And further I wonder how long before I have to think of transfering my current archived data to some yet newer technology and when do they lose compatibility with the present (like, when was the last time you connected a syquest disk?)?

So I'm wondering what other people who have dealt with this issue are doing in terms of backup hardware and related strategy for files they would like to keep for 50+ years.
Reply With Quote
  #2  
Old June 24th, 2006, 12:35 PM
Josh Liechty Josh Liechty is offline
Member
 
Join Date: May 2006
Location: Indiana, USA
Posts: 51
Default

Right now I'm facing the same problem, which was made worse this spring with the 15MB NEFs from my new camera (the ~3MB NEFs from my old D1 were much friendlier). My current strategy is to keep one copy on an internal hard disk on my desktop PC, another copy sent over the network to my PowerBook (but not much longer due to its rapidly filling 80GB hard disk), and a third copy burned onto a DVD-R. Obviously, this haphazard excuse for a strategy won't continue to work adequately in the future.

Having briefly considered tape backups and the related Iomega REV drives, I don't understand enough to figure out what to buy to make it work, and am discouraged by the high price. Either Blu-ray or HD-DVD might be a viable option, but the format "war" doesn't help concerns about future availability of one of the two, and neither seem quite ready or affordable enough yet for adoption as a large-scale backup solution.

These useless ramblings are really a "shortcut" to the subscribe-to-thread option, as I am eager to hear what photographers who have conquered this problem have to say about their hardware and methods.
__________________
Josh Liechty
Reply With Quote
  #3  
Old June 24th, 2006, 12:44 PM
Dierk Haasis Dierk Haasis is offline
Senior Member
 
Join Date: May 2006
Location: Hamburg (Germany)
Posts: 548
Default

1. Downloading from cards to internal HD.
2. Sorting, annotating etc.
3. Backing up to DVDs
4. Moving images from internal drive to external HD
5. Regular checks on HDs and DVDs
6. Probable re-saving to newer technologies.

I've given up on too much worrying after going through decades worth of slides and negatives to see what to scan and what not. Many of the 60s and 70s slides had been heavily damaged and I was lucky I could save so many through the wonders of digital (in this case: infrared cleaning and color correction). that were my father's and grandfather's photos. When I came to my own I scanned only those that couldn't be done again - one-time events withinn the family and historical breakthroughs like the fall of the Wall.

Should I lose both copies of a phozo, DVD and HD, I can shoot it again in many cases. Or I will not even notice it. Admittedly I threw away one negative I thought I could redo - just to find the object in question (a 70s fountain) to be torn down without my knowledge a few months earlier. Such is life ...
__________________
Dierk Haasis
[DH² Publishing]
Writing and Imaging

Nikon D2x, Nikon D200, Breeze DownloaderPro, Capture NX2, xMedia2, IDimager, Adobe Creative Suite 3
Reply With Quote
  #4  
Old June 24th, 2006, 12:49 PM
Eric Perlberg Eric Perlberg is offline
New Member
 
Join Date: Jun 2006
Posts: 7
Default

Well I'll add a bit about what I've learned so far.
1) There are large individual drives of 1tb which have an uncomfortably high failure rate. So I think it doesn't make sense to buy big drives to back up data as they're more failure prone and you loose more data if one does die.
2) There are large Network Area Storage (NAS) devices with replacable hard drives in some sort of RAID array, usually ATA drives or SATA drives which can then be used in RAID 1 for instant total backup or RAID 0 for speed or even RAID 5 for parity security (or just as a bunch of single disks). These replaceable drives can be bought at 1/3 the price of normal external drives and are the same as or better than and faster than and with bigger caches than the ones which get sold as good quality external hard drives by the name brands like Maxtor and Lacie. But NAS drives are slow except maybe at gigabit ethernet speeds and they require a large initial capital outlay with a break even point down the road depending on how much you shoot. I shoot about a gig a day 4 days a week on average. I figured it would be 2 years to break even for me
3) Then one has to question how long hard drives last. I learned today about gold plated DVDs which had a potential life of 100+ years but at 4gig a disk they would take 1.3 lifetimes to archive my images and they're expensive and the 100+ years sounds very theoretical without concrete testing.

Last edited by Eric Perlberg; June 24th, 2006 at 12:57 PM.
Reply With Quote
  #5  
Old June 24th, 2006, 02:41 PM
Sean DeMerchant Sean DeMerchant is offline
Inactive
 
Join Date: Jun 2006
Location: Whidbey Island, Washington State, USA
Posts: 735
Default

Quote:
Originally Posted by Eric Perlberg
Well I'll add a bit about what I've learned so far.
1) There are large individual drives of 1tb which have an uncomfortably high failure rate. So I think it doesn't make sense to buy big drives to back up data as they're more failure prone and you loose more data if one does die.
The largest drives on the market are 750 GB. The 1 TB models are actually two drives in RAID 0 which is not particularly reliable.
Quote:
Originally Posted by Eric Perlberg
2) There are large Network Area Storage (NAS) devices... But NAS drives are slow except maybe at gigabit ethernet speeds and they require a large initial capital outlay with a break even point down the road depending on how much you shoot. I shoot about a gig a day 4 days a week on average. I figured it would be 2 years to break even for me
Choosing NAS or direct attached storage will make little difference. USB 2.0 with a clean connection (nothing else connected) should work well. But Firewire 400 (IEEE 1394a), while theoretically slower, is often faster in practice due to differences in communication protocols.

The other thing with a USB/Firewire backup drive is that it is easily disconnected from both power and the computer which leaves it much safer from lighting strikes and power surges. Also, USB/Firewire drives are cheaper than NAS.

NAS has the bonus of RAID 5 to balance between protection and storage capacity. Be aware that RAID is not a substitute for keeping two separate copies as any RAID has a single point of failure (driver or hardware controller) that can still wipe it all out.

Quote:
Originally Posted by Eric Perlberg
3) Then one has to question how long hard drives last. I learned today about gold plated DVDs which had a potential life of 100+ years but at 4gig a disk they would take 1.3 lifetimes to archive my images and they're expensive and the 100+ years sounds very theoretical without concrete testing.
Hard drives outlast cheap media by years and have much faster access. Hard drives may cost slightly more, but if you factor in the time you spend creating and verifying DVDs and that cost can be quickly amortized. With the continually increasing disk capacity prices go down about 30%+ per GB per year so buying new bigger drives and moving files can be financially sound unless you have way more time than money.

Another solution to consider is magnetic tape. It was and still is a major industry backup standard method for reliability AFAIK. Tape solutions are not cheap as they are more of an enterprise than consumer product. But they can have high capacities (you get what you pay for).

A major thing to be aware of is that failures in disk to disk copy can fail and verifying your backups is vital to long term integrity no matter what method you utilize. To this end, I favor direct comparison, but even a file hash based comparison (CRC32, MD5, SHA1, ...) is better than nothing. A tool (free for non-commercial use) that makes this simple on Windows is CDCheck. Command line tools for the technically savvy would likely allow the usage of free tools and total automation.

With CD/DVD media I favor using parity files on the disks too. Take enough files to fill 2/3 of a disk, create 100% volume of parity files, and then burn two copies of the data with half the parity files on each disk. This allows recovery of all data when even moderately serious media failures occur.

In the end, I am currently looking at cheap external hard drive solution (buy the drive and external case and assemble) as buying pre-build external drives costs 5 or 10 times per GB what disk should cost.

some thoughts,

Sean
Reply With Quote
  #6  
Old June 24th, 2006, 03:37 PM
Eric Perlberg Eric Perlberg is offline
New Member
 
Join Date: Jun 2006
Posts: 7
Default

Quote:
In the end, I am currently looking at cheap external hard drive solution
Yeah I've been looking at this too since buying and installing myself an internal 7200 RPM 2.5" drive for my new MacBook and seeing how simple it is. A nice Western Digital Caviar 250gig 7200 RPM drive with a 16mb cache is only £50 while Lacie and Maxtor offer slower drives for 2.5 times the price and even more with triple interface connectors. And then label and store the disks.

Quote:
Choosing NAS or direct attached storage will make little difference
not sure we're talking about the same thing here but I should have typed Network Attached Storage. It's not direct connected, its an ethernet based multiuser environment. What attracted me to it was a) RAID 1 would let me set up two 250gb pairs and have fault tolerant backup and b) the hot swappable ability to do just as you suggest in your post, buy cheap internal SATAs and make them the archiving system.

Curious what others are doing. Anyone recommend a fast (firewire 800 or SATA), quiet, enclosure? Anyone have any data on failure rates of hard drives?
Reply With Quote
  #7  
Old June 24th, 2006, 03:48 PM
Don Lashier Don Lashier is offline
Senior Member
 
Join Date: Jun 2006
Location: Oregon Coast
Posts: 745
Default

> Anyone have any data on failure rates of hard drives?

My consumer EIDE drives seem to have an average life of about two years. With server grade SCSI (I currently have 44 in operation, most for 5 years or more), I've had exactly two failures in 10 years. Note that it's not only drive quality that matters but proper cooling.

- DL
__________________
Don Lashier
www.lashier.com
- old enough to know better - but I do it anyway
Reply With Quote
  #8  
Old June 24th, 2006, 04:01 PM
Sean DeMerchant Sean DeMerchant is offline
Inactive
 
Join Date: Jun 2006
Location: Whidbey Island, Washington State, USA
Posts: 735
Default

Quote:
Originally Posted by Eric Perlberg
Yeah I've been looking at this too since buying and installing myself an internal 7200 RPM 2.5" drive for my new MacBook and seeing how simple it is. A nice Western Digital Caviar 250gig 7200 RPM drive with a 16mb cache is only £50 while Lacie and Maxtor offer slower drives for 2.5 times the price and even more with triple interface connectors. And then label and store the disks.
Currently, I am maxed out internally with 7 drives plus DVDRW.

In the States one get 250-300 GB PATA drives for slightly less than $100 USD online from reputable dealers. Maxtor DiamondMax 10 250 GB SATA drives can be found for $75 USD in OEM state (no screws or cables included and a shorter warranty IIRC).

The drives are easy to find, but I want to find a reliable external cases for PATA or SATA drives with Firewire and USB connectors (NAS ethernet would be nice too, but adds to cost). The problem is I have not noted any quality reviews of said products to help make a choice.

Quote:
Originally Posted by Eric Perlberg
not sure we're talking about the same thing here but I should have typed Network Attached Storage. It's not direct connected, its an ethernet based multiuser environment. What attracted me to it was a) RAID 1 would let me set up two 250gb pairs and have fault tolerant backup and b) the hot swappable ability to do just as you suggest in your post, buy cheap internal SATAs and make them the archiving system.
Same thng, NAS is just attached via a network rather than a direct connection. Commercial solutions often step up to the next level and use SAN (Storage Area Network) with very fast and expensive networking gear so that servers may share storage and allow easy expansion of storage.

With NAS things range from appliances (portable RAID units) to full blown systems.

Another option with NAS is to use an old computer with something like http://www.freenas.org/ (early and free version of a web based unix system admistration for NAS) and then max out the drives.

My issue with RAID 1, 3, 5, 6, 10, 0+1, 50, ... is that they still has single points of failure. They just has less single points of failure than a single drive.

Quote:
Originally Posted by Eric Perlberg
Curious what others are doing. Anyone recommend a fast (firewire 800 or SATA), quiet, enclosure? Anyone have any data on failure rates of hard drives?
eSATA is an option. It is just sturdier connector, shielded cabling, and it is hot pluggable (no need to reboot). eSATA may be worth exploring if cost effective solutions exist. As this technology is still emerging, todays viewpoint may be very out of date in 6 months.

eSATA over a PCI based controller may have limited value unless every PCI slot is on its own bus (server style motherboards). A PCI-Express x4 eSATA controller would have serious appeal.

Firewire and USB are more likely to be available on others systems.

Firewire 800 (IEEE 1394b) is less common. I have never seen any reviews noting it helps with single drive. It may be more useful with chained drives.

Myself, I cannot countenance spending more than $500 USD per TB of storage.

I would also like to hear success stories for large 0.5+ TB of data storage.

some thoughts,

Sean
Reply With Quote
  #9  
Old June 24th, 2006, 04:07 PM
Josh Liechty Josh Liechty is offline
Member
 
Join Date: May 2006
Location: Indiana, USA
Posts: 51
Default

There are tons of enclosures on Newegg, but the reviews there border on useless, and I would also like to hear what's working for people who use external hard drives already.

Some cost data that I found while reading about Quantum's Govault system (removable hard disk cartridges, like Iomega's Rev system) might be of interest. A 120GB Govault cartridge (excluding the drive) is US$2.16 per GB. 400GB external hard disks that come in an enclosure tend to be around $0.50/GB, with 250GB external drives costing approximately $0.44/GB. If one were to take a 300GB drive sans enclosure (~$99) and put it in an enclosure (~$30), it wouldn't be much different than a pre-built 250GB external drive, at $0.43/GB. This is just a sampling of a few options on Newegg, so it's quite possible that you could do something for less than what I've estimated here. These results do call into question, however, my thoughts that a removable disk cartridge system's advantages outweigh the cost penalty.
__________________
Josh Liechty
Reply With Quote
  #10  
Old June 24th, 2006, 07:04 PM
Ray West Ray West is offline
Senior Member
 
Join Date: Jun 2006
Location: UK - Somerset
Posts: 1,703
Default

There have been many removeable cartridge systems over the years, but they never seem to stay around more than a couple of years or so. I have used Panasonic md system, iomega and syquest. All of the drives gradually failed, some manufacturers went bust, and I have various cartridges that are of no use.

I think the best route at the moment is external hdds, put into your own usb cases. If you buy seagate bare drives, they have a five year warranty. If you buy a pre-assembled usb drive, containing a seagate drive, it carries the case/system guarantee, which will not be five years. Seagate do not offer a warranty to end users for their oem drives, at least not in the UK. Some external cases have cooling fans, and some keep the drive powered up all the time.

However, it is not that much more expensive to use any old pc on a network, loaded with internal drives, possibly a raid system, compared to a number of 'quality' external usb drive cases. SCSI drives are generally more reliable, but tend to be more expensive for the same capacity.

The more important the data is, the more copies you need to keep.

Best wishes
Reply With Quote
  #11  
Old June 28th, 2006, 02:15 PM
Eric Perlberg Eric Perlberg is offline
New Member
 
Join Date: Jun 2006
Posts: 7
Default

Well at this point... I just purchased two 300gb D2 triple interface LaCie drives. I decided to do this instead of buying enclosures because I did some research on one enclosure company ICY BOX which everyone in the UK seems to carry. Some people were having real problems with getting their drives to show up, others weren't, so even though theory says things should be straight forward, practice shows that Murphy's Law and unanticipated gotchas operates here as well as everywhere else. For example, on a Mac at least, to have a bootable hard drive using Firewire 400, you need a recent chip set. The majors seem to all use appropriate ones but not all firewire 400 chipsets are compatible and enclosure manufacturers frequently don't say anything about functioning as startup disks on a Mac (small market for them I guess, except maybe Macally enclosures) . The issue of noise came up at another point. Without being able to verify the noise of either a fan (IcyBox doesn't use one) or the clutter of the drive mechanism (specific to the drive not the enclosure) and since I don't have time to sort through all kinds of non-photographic issues, I took the fast/easy way out.

I'm going to try to use the new 300gb drives using Macintosh's built in RAID software. I should be able to use RAID 1 thus eliminating (fingers crossed) the need to back up through software or manually.

I do think that long term archiving is an under discussed weakness of digital at this point though the future will undoubtably bring something more fit to purpose.

Last edited by Eric Perlberg; June 28th, 2006 at 02:54 PM.
Reply With Quote
  #12  
Old June 28th, 2006, 10:11 PM
Sean DeMerchant Sean DeMerchant is offline
Inactive
 
Join Date: Jun 2006
Location: Whidbey Island, Washington State, USA
Posts: 735
Default

Quote:
Originally Posted by Eric Perlberg

I'm going to try to use the new 300gb drives using Macintosh's built in RAID software. I should be able to use RAID 1 thus eliminating (fingers crossed) the need to back up through software or manually.
RAID 1 is still subject to singe points of failure:
  1. A virus deleting one file will delete both copies.
  2. A driver failure can corrupt both drives simultaneously.
  3. User error deleting one file will delete both copies.
  4. A lighting strike/power surge can wipe both out simultaneously.
  5. ...
In short, a pair of external drives where one is always unplugged from the computer and the wall is much safer. But a single fire/earthquake/flood/hurricane/tornado/... can wipe out both drives.

All RAID 1 covers is hard drive failure.

enjoy,

Sean
Reply With Quote
  #13  
Old July 5th, 2006, 08:02 PM
Rob Peterson Rob Peterson is offline
Member
 
Join Date: Jul 2006
Location: North Texas, USA
Posts: 38
Default

Quote:
Originally Posted by Sean DeMerchant
My issue with RAID 1, 3, 5, 6, 10, 0+1, 50, ... is that they still has single points of failure. They just has less single points of failure than a single drive.
Seems to me that a RAID 5 or 10 protects against a drive failure, in my experience the most common failure. Yes, a controller failure could destroy the data, but such a failure seems quite rare.
Quote:
Myself, I cannot countenance spending more than $500 USD per TB of storage.

I would also like to hear success stories for large 0.5+ TB of data storage.

Sean
Three years ago I installed in machines at two different locations RAID 5 arrays based on SCSI drives and Adaptec RAID controllers. Drives in both arrays failed, but I never lost access to the data. In one case the controller rebuilt the data on a hot spare before I could buy and install the replacement drive. Recovery from the failures involved shutting down the machine, replacing the failed drive, and restarting the machine. The controller rebuilt the failed drive's data while continuing to support data access.

Last year I added to one of the machines with a SCSI RAID an Adaptec RAID controller with 8 SATA ports, and connected 6 250GB drives. The 8 port controller solves the problem of how to connect more than 4 SATA drives, although case space becomes an issue. I spent just over $400 for the SATA RAID controller, and about $100 each for the drives. I now have 0.9TB of SATA RAID5 at $1,000/TB, but those are 12 month old prices. Today I'd build a 2TB (net) RAID 5 array for $420 + 8 300GB drives at $80/each, or $420 + $640 = $1,100/2TB, or start smaller with a higher per TB cost and incrementally add drives as I need more space. Note that adding another drive potentially expands the logical volume rather than becoming a new logical volume.

I consider my installation a success, and I intend to continue growing my SATA RAID solution.

My outstanding issue is upgrading my backup solution. DDS4 tape just doesn't work for TB storage! I'm considering a high capacity tape drive (LTO2 at 200GB/tape?) or a tape library.

Bob
Reply With Quote
  #14  
Old July 10th, 2006, 04:52 AM
Jason Anderson Jason Anderson is offline
Member
 
Join Date: May 2006
Location: SC
Posts: 53
Default

Quote:
Originally Posted by Sean DeMerchant
RAID 1 is still subject to singe points of failure:
  1. A virus deleting one file will delete both copies.
  2. A driver failure can corrupt both drives simultaneously.
  3. User error deleting one file will delete both copies.
  4. A lighting strike/power surge can wipe both out simultaneously.
  5. ...
In short, a pair of external drives where one is always unplugged from the computer and the wall is much safer. But a single fire/earthquake/flood/hurricane/tornado/... can wipe out both drives.

All RAID 1 covers is hard drive failure.

enjoy,

Sean
1. Fair point on the virus, but if you have the know how to set up a RAID array, you should also have the know how to implement both software and hardware firewalls, as well as the use of a reliable AV package.

2. A driver failure will cause a drive not to be recognized by the RAID card or the OS, so the drive really isn't corrupted, just inaccessible. Reinstalling the driver (often by simply rebooting) solves this problem.

3. If you delete a file accidentally, the backups that are not part of your active drives should address that problem. I won't go into a person adept enough to install RAID, but still inattentive enough to delete files...

4. The lightning argument will hold for ANY setup that does not have surge protection, so in this respect RAID 1 is no different than any other array, so I fail to see the point of this argument. The same point does hold that held in #'s 1 and 3 though - if you know enough to RAID, you should also know enough to include surge protection and UPS solutions strategies.

5. What else....? :)

An external drive relies on the person to actively make copies of everything. If that copy isn't made, there is no redundancy...RAID by definition, once set up, is redundant...Redundant Array of Independant Disks...
__________________
Jason Anderson
EOS 350D, kit, 70-300 Tamron and a sack to carry it
Regional affiliation: www.scphotogs.com
Reply With Quote
  #15  
Old July 10th, 2006, 05:36 AM
Eric Perlberg Eric Perlberg is offline
New Member
 
Join Date: Jun 2006
Posts: 7
Default

One issue which has to be addressed by everyone contemplating serious backup is just how much is your data worth and where to draw the spending line. As I said in my inital post, data backup risk management is a slippery slope as one can continually come up with potential threats to one's data requiring ever more extreme solutions with ever increasing cost implications.

I don't know if its even possible to do but it might be a useful idea to develop some sort of heuristic which helps an individual decide where to draw the line. There are amateur photographers who face one sort of risk, professional fashion, wedding, glamour, news, etc. photographers who have to face another set of questions and then there is the professional artist working in photography who face yet a different set of issues.
Reply With Quote
  #16  
Old July 29th, 2006, 01:59 AM
Andreas Kanon Andreas Kanon is offline
Member
 
Join Date: Jul 2006
Location: Belleville, Michigan
Posts: 40
Default

Right now i just mirror the data onto two different hard drives.
I am waiting for Blu Ray to start to ship in volume.
The specs i have seen on some blue ray readers/writers they can read regular cd and dvds just fine.
On one single Blu Ray disk you can get 25 or 50 Gb of data on a disk the same size as a DVD.
At least for me that will put a temporary end to my problems of backing up to a long term storage solution.
__________________
www.kanonphoto.com
Reply With Quote
  #17  
Old July 29th, 2006, 02:08 AM
Eric Perlberg Eric Perlberg is offline
New Member
 
Join Date: Jun 2006
Posts: 7
Default

I've been thinking that too with all the talk suddenly about Blue Ray. It will be interesting to see what the longevity of those disks prove to be.
Reply With Quote
  #18  
Old July 29th, 2006, 02:14 AM
Andreas Kanon Andreas Kanon is offline
Member
 
Join Date: Jul 2006
Location: Belleville, Michigan
Posts: 40
Default

Quote:
Originally Posted by Eric Perlberg
I've been thinking that too with all the talk suddenly about Blue Ray. It will be interesting to see what the longevity of those disks prove to be.
Agreed it will be interresting to see.
Personally i will be making 2 copies, one in the bank and one at home.
__________________
www.kanonphoto.com
Reply With Quote
  #19  
Old July 29th, 2006, 09:21 AM
Gary Ayala Gary Ayala is offline
Senior Member
 
Join Date: Jul 2006
Location: SoCal
Posts: 548
Default Boxes

About a year or so ago, I found an old box filled with equally old photos. The snaps were more than two decades old and very well preserved (since they were never exposed to light). I've been scanning them and tossing them on my wedsite. Using that experience as a lesson ... I suggest printing your best stuff and boxing it. (Not very romantic, or quick ... but a solution nonetheless.)

My old snaps are here:

http://www.garyayala.smugmug.com/gallery/665619
__________________
Unsharp at Any Speed
My Snaps are here:
www.garyayala.smugmug.com
Reply With Quote
  #20  
Old September 1st, 2006, 11:54 PM
Nikolai Sklobovsky Nikolai Sklobovsky is offline
Senior Member
 
Join Date: Aug 2006
Location: Thousand Oaks, CA, USA
Posts: 1,181
Default Glad I found this!

I was about to post a question, but decided to search first - and, man, was I rewarded by this thread...:-)

My HDD is almost filled up. When I got it less than 6 months ago I thought 250Gb will serve me 2-3 years... No such luck. RAW files use the space fast...;-)

DVDs do not seem to make the cut, since I'd have to burn one every week or so. Let alone you'd have to use a rather expensive media for the archiving purposes, since the el cheapo $.50 blanks won't last even 5 years.

External HDDs... Well, I have my doubts, too. Unless you keep them spinning they can deform. They can "the magnetic charge". Anyway, I have no reason to believe they will sit quietly in the closet for 5-10 years and then with a single power switch flip they will spring back to life..

Tape was kinda industry-standard for the backups for decades. Although I did see my good share of bad tapes to not trust them either. And they seem to be still quite expensive, albeit recently some of the devices became available for the low-end consumer market.

BluRay looks promising, but it's not here yet price-wise.

So I hope for the BluRay (or HDVD) to come in 2-3 years. The question is - how to survive until it's here.
My current line of thoughts - external 400-500GB HDD with compression turned on, which should make it into 600-800Gb. That would hopefully give me a piece of mind until the high-capacity optical writable media is here, and after that it may still serve well as a secondary storage device.
__________________
Nikolai
Sr.Software Engineer
PhotoSoCal, Digital Grin, LA Shooters, NAPP, PPA
Reply With Quote
  #21  
Old September 2nd, 2006, 12:25 AM
Asher Kelman Asher Kelman is offline
OPF Owner/Editor-in-Chief
 
Join Date: Apr 2006
Posts: 34,974
Default

Blue Ray DVD burned with an inorganic chemical process is about to be available. This to me will make the best storage medium when the disks are stored vertical in in archival bags in a steel cabinet. The Blue RAy may turn out to be reliable for a decade or more.

Hard drives need to be examined for latent errors at least once a year and IMHO, backed up to new media every 5 years.

asher
Reply With Quote
  #22  
Old September 2nd, 2006, 12:49 AM
Sean DeMerchant Sean DeMerchant is offline
Inactive
 
Join Date: Jun 2006
Location: Whidbey Island, Washington State, USA
Posts: 735
Default

Quote:
Originally Posted by Asher Kelman
Hard drives need to be examined for latent errors at least once a year and IMHO, backed up to new media every 5 years.
Unless there is a massive shift in the current evolution of HDD pricing, simply backing up to a new drive once a year (double the storage for the same price) would seem reasonable to me. My computer is currently maxed out* at ~1.6TB and I am within a few months or running out of storage. I am seriously considering buying 3.5" external firewire enclosures and 300 GB drives with the intent of buying larger drives as the sweet spot on the price curve changes.

But, using anything but HDD is hard for me to countainence as they are more reliable than all but tape media and the human cost in terms of time of burning CD/DVD/... is huge.

As to latent errors, there are two solutions.

1) Annual copies of pairs of drives to other pairs of drives plus running a disk check for bad sectors.

2) Parity files. Creating 100% parity data and then copying 1/2 the parity to each of a pair of disks increases storage usage by 33% while greatly increasing redundancy. Albeit, this too has great cost in terms of user time.

I personally use parity files for anything shifted to DVD+R (and once CDR which is prohibitively small now).

enjoy,

Sean


* Unless I replace drives with larger ones.
Reply With Quote
  #23  
Old September 2nd, 2006, 12:54 AM
Asher Kelman Asher Kelman is offline
OPF Owner/Editor-in-Chief
 
Join Date: Apr 2006
Posts: 34,974
Default

Hi Sean,

Love that flower and (is it a) fly!!

I'd get SATA, not more firewire as the Mac doesn't like too many for some reason and may give Kernel panics.

Now what do you use to check your disk media and how do you create your parity files? Did you ever say what OS you use?

Asher
Reply With Quote
  #24  
Old September 2nd, 2006, 02:20 AM
Sean DeMerchant Sean DeMerchant is offline
Inactive
 
Join Date: Jun 2006
Location: Whidbey Island, Washington State, USA
Posts: 735
Default

Quote:
Originally Posted by Asher Kelman

Love that flower and (is it a) fly!!
I am assuming your are talking about the digger bee on the gum plant in the other thread. Thanks. :o) Or did I miss something.

Truth be told, I have shifted my shooting style to try and get additional shots for http://www.bugguide.net to get hlep on IDing the beautiful little ones and to add to their historical data.
Quote:
Originally Posted by Asher Kelman
I'd get SATA, not more firewire as the Mac doesn't like too many for some reason and may give Kernel panics.
My issue here is that I am looking at pairs of drive of which no more than one will ever be connnect to the system at once. If I go SATA, then it will be a $800 PCI-Express RAID 5/6 card and a stack of external exclosures do the lack of ports on hot swapable E-SATA cards. But truth be told, what I want is offline storage and Firewire and USB compatible (i.e., both and not either) drives are less expensive and can be disconnected from the power mains making it easy to store an archival copy of a full disk at a friend or family members (or a bank safe deposit box) to increase earthquake/flood/volcanic data survival.
Quote:
Originally Posted by Asher Kelman


Now what do you use to check your disk media and how do you create your parity files? Did you ever say what OS you use?
I would simply run checkdisk in XP as I run XP. And in practice, I would run checkdisk on the new and old disk, copy the files over, and then compare the files bit for bit to verify the copy.

I would like a Mac as I run bash as my shell anyway and I truly broke my teeth on computer systems using unix. But XP is cheaper and almost reliable. If I could make a business case for buyig a Mac Pro, I would. But I cannot. Hence, day to day at the command line I run bash shell under cygwin and ignore the MSFT nastiness under the hood.

But practically speaking, I have dual cores and 3 GB of RAM so performance deltas are minor until Adobe releases an x86 PS binary anyway

In short, I use XP, but I am OS agnostic so long as I can run my tools.

enjoy,

Sean (who is falling asleep at the keyboard)
Reply With Quote
  #25  
Old September 2nd, 2006, 10:26 AM
Brian Lowe Brian Lowe is offline
Member
 
Join Date: May 2006
Location: Hollywood CA
Posts: 175
Default Jungledisk

As an computer network administrator I am always concerned about data storage and disaster recovery. So for storing my RAW files I use Jungle Disk it uses Amazon.com's S3™ Storage. Data is stored at multiple Amazon.com data centers around the country. Plus it very cheep .15 cents a gigabyte.

Living in Southern California we have earthquakes so, I like the idea of storing my photo data files in multiple data centers though out the country.

You may want to look into this I love having redundancy in data storage.



You have to ask yourself whats your photo data worth to you?


Brian
__________________
=========================================
Brian,
for the home of "Lowe Quality" Photography
click here=>http://www.brianlowe.net


Reply With Quote
  #26  
Old September 2nd, 2006, 12:06 PM
Nikolai Sklobovsky Nikolai Sklobovsky is offline
Senior Member
 
Join Date: Aug 2006
Location: Thousand Oaks, CA, USA
Posts: 1,181
Default

Brian,

can you please provide a bit more details on this JungleDisk deal? Please??

TIA!
__________________
Nikolai
Sr.Software Engineer
PhotoSoCal, Digital Grin, LA Shooters, NAPP, PPA
Reply With Quote
  #27  
Old September 2nd, 2006, 01:02 PM
Brian Lowe Brian Lowe is offline
Member
 
Join Date: May 2006
Location: Hollywood CA
Posts: 175
Default

Hi Nik & others

Take a look for yourself and decide if it's for you.
All of the details are right here ==> http://www.jungledisk.com/
__________________
=========================================
Brian,
for the home of "Lowe Quality" Photography
click here=>http://www.brianlowe.net


Reply With Quote
  #28  
Old September 2nd, 2006, 01:52 PM
Sean DeMerchant Sean DeMerchant is offline
Inactive
 
Join Date: Jun 2006
Location: Whidbey Island, Washington State, USA
Posts: 735
Default

Quote:
Originally Posted by Brian Lowe
Hi Nik & others

Take a look for yourself and decide if it's for you.
All of the details are right here ==> http://www.jungledisk.com/
It is very expensive.

Consider:

initial upload of 250 GB -> $50.00
1 month of storage plus an upload of 25 GB -> $42.50
1 month of storage plus an upload of 25 GB -> $46.25
1 month of storage plus an upload of 25 GB -> $50.00
1 month of storage plus an upload of 25 GB -> $53.75
1 month of storage plus an upload of 25 GB -> $57.50
1 month of storage plus an upload of 25 GB -> $61.25
1 month of storage plus an upload of 25 GB -> $65.00
1 month of storage plus an upload of 25 GB -> $68.75
1 month of storage plus an upload of 25 GB -> $72.50
1 month of storage plus an upload of 25 GB -> $76.25
1 month of storage plus an upload of 25 GB -> $80.00
1 month of storage plus an upload of 25 GB -> $83.75

Yielding $807.50 USD for the first years of storage if you shoot a moderate 25 GB of new images per month. Over a period of years this will add up. And since I can get a Firewire/USB case and a 300GB drive for $175 USD with shipping I can get 600 GB of redundant (two copies) online storage for $700. Plus, the slower USB tranfer rate far exceeds my DSL upload speed meaning that the network backup is really slow. Add in that you are using beta software to achieve this storage and I cannot see the value proposition as periodically mailing a drive to a friend or relative would still be cheaper.

And if you shoot more than 25 GB a month this would be very expensive. Though if you can make a business case for it the distributed nature has disaster recovery value. But if your credit card expires the month the disaster hits and they simply delete your data as you cannot contact them to update payment info, you get hit again.

my $0.02,

Sean
Reply With Quote
  #29  
Old September 3rd, 2006, 12:35 PM
Alan T. Price Alan T. Price is offline
Member
 
Join Date: Jun 2006
Location: Perth, Western Australia
Posts: 101
Default

In my opinion we are pretty well stuffed because there is no such thing as long term storage in this industry. 50 years is a pipe dream. 5 years is really stretching it. 2-3 years is likely. Tapes and discs (both optical and magnetic) all deteriorate. The only way to prevent data loss is to have a strict data management regime that will include documentation in case you should forget a step, and multiple on- and off-line and on- and off-site backups. But those individual back-up components need to be manged too.

Many of the posts to this thread have mentioned SATA drives. Well, consider this: what happened to PATA ? It's almost vanished in the space of a few months. SATA will go the same way one day. So will any media type. It's an inevitability of a commercially-oriented business world. The hardware and software manufacturers are looking after their interests; not mine, and not yours.

Even a well stocked SATA system with plenty of spare capacity will eventually need additional or replacement drives that will not fit or will not connect. Then you are up for another new system. Now that's a terrifying prospect because by then you will not necessarily be able to copy your data from the old system to the new system. All of those SATA drives in the dedicated SATA storage system will be rendered useless as soon as the dedicated system controller itself dies, and then you find that a new one cannot be found, and your SATA drives won't fit the shiny new PC you have by then.

Similarly, in a years time none of your PCI cards will find a new home. None of your 16-bit system software will find an operating system. You might think "so what?" but the software we have for some of the older drives including tape is 16 bit. Even some of the 32-bit software died when Windows XP went to SP2, but of course other new software won't work without SP2. My favourite automatic backup utility no longer works and is no longer supported since Symantec bought out PowerQuest. Thank you Symantec, for nothing. There are just too many examples of this for us to believe it won't happen again.

Even the comfort of having a warranty is lost when the manufacturer has gone bust - warranties no longer apply.

I don't know what the solution is but I'm inclined to think that it needs to be low-tech and cheap and to have only just enough capacity for the short term. It also needs to be off-line most of the time to prevent runaway software from attacking it when it attackes the rest of your PC. [I had something try to corrupt all .jpg files on my PC despite having the latest AV and firewall facilities. It corresponded to installing Irfanview and went away when I uninstalled it, and yet zillions of people have used Irfanview without a problem. Lucky me, I guess, but I got the images back from my off-line backups. Careful me :)]

Condider for example a pair of large-ish RAID 1 drives in a simple external controller so that...
(1) a separate PC and operating system is not required to drive it,
(2) if a drive dies you may be able to replace it,
(3) if the controller dies you may be able to replace that,
(4) if either the drive or the controller dies and cannot be replaced you can buy a whole new set of different technology or brand relatively cheaply
(5) when new technology comes out (such as SATA replacing PATA) you can buy a whole new set, with bigger capacity, before the old system has a chance to die,
(6) it is relatively affordable to replace it when required,
(7) with any luck it will still work when you upgrade to a new PC (and if it doesn't then there's time to replace it before you dispose of the old PC).

The longest-lasting image backups I've had to date are, believe it or not, on good old film. What does that say for the digital photography/computer industry ?

The bottom line, in my opinion, is that It is important that the backup solution must not be allowed to outlive the technology or the computer/software support or the manufacturer or even the manufacture, or else you will eventually find your backup is compromised, and it may happen much sooner than later.
__________________
Alan (Perth, Western Australia)

Canon 1Ds2, 1D2, 1V.
Far too many lenses.
Reply With Quote
  #30  
Old September 3rd, 2006, 01:47 PM
Nikolai Sklobovsky Nikolai Sklobovsky is offline
Senior Member
 
Join Date: Aug 2006
Location: Thousand Oaks, CA, USA
Posts: 1,181
Default

Alan,

great post! Thank you very much!

Now I have a favor to ask: can you provide a sample configuration for your suggested solution? Something capable of storing, say, 1-2 Tb for the next 2-3 years?

Ta, mate!
__________________
Nikolai
Sr.Software Engineer
PhotoSoCal, Digital Grin, LA Shooters, NAPP, PPA
Reply With Quote
Reply

Bookmarks

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are Off
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -7. The time now is 08:00 PM.


Posting images or text grants license to OPF, yet © of such remain with its creator. Still, all assembled discussion © 2006-2017 Asher Kelman (all rights reserved) Posts with new theme or unusual image might be moved/copied to a new thread!