Longer term storage

[Eric Perlberg]

I'm just getting to the point where I need to develop a serious storage strategy rather than a kind of adhoc affair. At the moment I simply back up both RAW images and processed images onto 2 internal hard drives. When I get 250 gig or so of images, I buy a cheap usb2 hard drive, copy RAW and Processed files to the new drive, make sure they're really there, then pack it all up in its original packing with power supply and cable and put it in the closet. Then I clear the data from the internal drives. I could easily convince myself to back this data up twice because drives fail. But I can see that as a slippery slope leading me to remortgaging, etc. And further I wonder how long before I have to think of transfering my current archived data to some yet newer technology and when do they lose compatibility with the present (like, when was the last time you connected a syquest disk?)?

So I'm wondering what other people who have dealt with this issue are doing in terms of backup hardware and related strategy for files they would like to keep for 50+ years.

[Josh Liechty]

Right now I'm facing the same problem, which was made worse this spring with the 15MB NEFs from my new camera (the ~3MB NEFs from my old D1 were much friendlier). My current strategy is to keep one copy on an internal hard disk on my desktop PC, another copy sent over the network to my PowerBook (but not much longer due to its rapidly filling 80GB hard disk), and a third copy burned onto a DVD-R. Obviously, this haphazard excuse for a strategy won't continue to work adequately in the future.

Having briefly considered tape backups and the related Iomega REV drives, I don't understand enough to figure out what to buy to make it work, and am discouraged by the high price. Either Blu-ray or HD-DVD might be a viable option, but the format "war" doesn't help concerns about future availability of one of the two, and neither seem quite ready or affordable enough yet for adoption as a large-scale backup solution.

These useless ramblings are really a "shortcut" to the subscribe-to-thread option, as I am eager to hear what photographers who have conquered this problem have to say about their hardware and methods.

[Dierk Haasis]

1. Downloading from cards to internal HD.
2. Sorting, annotating etc.
3. Backing up to DVDs
4. Moving images from internal drive to external HD
5. Regular checks on HDs and DVDs
6. Probable re-saving to newer technologies.

I've given up on too much worrying after going through decades worth of slides and negatives to see what to scan and what not. Many of the 60s and 70s slides had been heavily damaged and I was lucky I could save so many through the wonders of digital (in this case: infrared cleaning and color correction). that were my father's and grandfather's photos. When I came to my own I scanned only those that couldn't be done again - one-time events withinn the family and historical breakthroughs like the fall of the Wall.

Should I lose both copies of a phozo, DVD and HD, I can shoot it again in many cases. Or I will not even notice it. Admittedly I threw away one negative I thought I could redo - just to find the object in question (a 70s fountain) to be torn down without my knowledge a few months earlier. Such is life ...

[Eric Perlberg]

Well I'll add a bit about what I've learned so far.
1) There are large individual drives of 1tb which have an uncomfortably high failure rate. So I think it doesn't make sense to buy big drives to back up data as they're more failure prone and you loose more data if one does die.
2) There are large Network Area Storage (NAS) devices with replacable hard drives in some sort of RAID array, usually ATA drives or SATA drives which can then be used in RAID 1 for instant total backup or RAID 0 for speed or even RAID 5 for parity security (or just as a bunch of single disks). These replaceable drives can be bought at 1/3 the price of normal external drives and are the same as or better than and faster than and with bigger caches than the ones which get sold as good quality external hard drives by the name brands like Maxtor and Lacie. But NAS drives are slow except maybe at gigabit ethernet speeds and they require a large initial capital outlay with a break even point down the road depending on how much you shoot. I shoot about a gig a day 4 days a week on average. I figured it would be 2 years to break even for me
3) Then one has to question how long hard drives last. I learned today about gold plated DVDs which had a potential life of 100+ years but at 4gig a disk they would take 1.3 lifetimes to archive my images and they're expensive and the 100+ years sounds very theoretical without concrete testing.

[Sean DeMerchant]

Well I'll add a bit about what I've learned so far.
1) There are large individual drives of 1tb which have an uncomfortably high failure rate. So I think it doesn't make sense to buy big drives to back up data as they're more failure prone and you loose more data if one does die.

The largest drives on the market are 750 GB (http://www.seagate.com/products/consumer_electronics/db35series.html). The 1 TB models are actually two drives in RAID 0 which is not particularly reliable.

2) There are large Network Area Storage (NAS) devices... But NAS drives are slow except maybe at gigabit ethernet speeds and they require a large initial capital outlay with a break even point down the road depending on how much you shoot. I shoot about a gig a day 4 days a week on average. I figured it would be 2 years to break even for me


Choosing NAS or direct attached storage will make little difference. USB 2.0 with a clean connection (nothing else connected) should work well. But Firewire 400 (IEEE 1394a), while theoretically slower, is often faster in practice due to differences in communication protocols.

The other thing with a USB/Firewire backup drive is that it is easily disconnected from both power and the computer which leaves it much safer from lighting strikes and power surges. Also, USB/Firewire drives are cheaper than NAS.

NAS has the bonus of RAID 5 to balance between protection and storage capacity. Be aware that RAID is not a substitute for keeping two separate copies as any RAID has a single point of failure (driver or hardware controller) that can still wipe it all out.


3) Then one has to question how long hard drives last. I learned today about gold plated DVDs which had a potential life of 100+ years but at 4gig a disk they would take 1.3 lifetimes to archive my images and they're expensive and the 100+ years sounds very theoretical without concrete testing.

Hard drives outlast cheap media by years and have much faster access. Hard drives may cost slightly more, but if you factor in the time you spend creating and verifying DVDs and that cost can be quickly amortized. With the continually increasing disk capacity prices go down about 30%+ per GB per year so buying new bigger drives and moving files can be financially sound unless you have way more time than money.

Another solution to consider is magnetic tape. It was and still is a major industry backup standard method for reliability AFAIK. Tape solutions are not cheap as they are more of an enterprise than consumer product. But they can have high capacities (you get what you pay for).

A major thing to be aware of is that failures in disk to disk copy can fail and verifying your backups is vital to long term integrity no matter what method you utilize. To this end, I favor direct comparison, but even a file hash based comparison (CRC32, MD5, SHA1, ...) is better than nothing. A tool (free for non-commercial use) that makes this simple on Windows is CDCheck (http://www.elpros.si/CDCheck/). Command line tools for the technically savvy would likely allow the usage of free tools and total automation.

With CD/DVD media I favor using parity files (http://www.quickpar.org.uk/) on the disks too. Take enough files to fill 2/3 of a disk, create 100% volume of parity files, and then burn two copies of the data with half the parity files on each disk. This allows recovery of all data when even moderately serious media failures occur.

In the end, I am currently looking at cheap external hard drive solution (buy the drive and external case and assemble) as buying pre-build external drives costs 5 or 10 times per GB what disk should cost.

some thoughts,

Sean

[Eric Perlberg]

In the end, I am currently looking at cheap external hard drive solution

Yeah I've been looking at this too since buying and installing myself an internal 7200 RPM 2.5" drive for my new MacBook and seeing how simple it is. A nice Western Digital Caviar 250gig 7200 RPM drive with a 16mb cache is only £50 while Lacie and Maxtor offer slower drives for 2.5 times the price and even more with triple interface connectors. And then label and store the disks.

Choosing NAS or direct attached storage will make little difference not sure we're talking about the same thing here but I should have typed Network Attached Storage. It's not direct connected, its an ethernet based multiuser environment. What attracted me to it was a) RAID 1 would let me set up two 250gb pairs and have fault tolerant backup and b) the hot swappable ability to do just as you suggest in your post, buy cheap internal SATAs and make them the archiving system.

Curious what others are doing. Anyone recommend a fast (firewire 800 or SATA), quiet, enclosure? Anyone have any data on failure rates of hard drives?

[Don Lashier]

> Anyone have any data on failure rates of hard drives?

My consumer EIDE drives seem to have an average life of about two years. With server grade SCSI (I currently have 44 in operation, most for 5 years or more), I've had exactly two failures in 10 years. Note that it's not only drive quality that matters but proper cooling.

- DL

[Sean DeMerchant]

Yeah I've been looking at this too since buying and installing myself an internal 7200 RPM 2.5" drive for my new MacBook and seeing how simple it is. A nice Western Digital Caviar 250gig 7200 RPM drive with a 16mb cache is only £50 while Lacie and Maxtor offer slower drives for 2.5 times the price and even more with triple interface connectors. And then label and store the disks.


Currently, I am maxed out internally with 7 drives plus DVDRW.

In the States one get 250-300 GB PATA drives for slightly less than $100 USD online from reputable dealers. Maxtor DiamondMax 10 250 GB SATA drives can be found for $75 USD in OEM state (no screws or cables included and a shorter warranty IIRC).

The drives are easy to find, but I want to find a reliable external cases for PATA or SATA drives with Firewire and USB connectors (NAS ethernet would be nice too, but adds to cost). The problem is I have not noted any quality reviews of said products to help make a choice.


not sure we're talking about the same thing here but I should have typed Network Attached Storage. It's not direct connected, its an ethernet based multiuser environment. What attracted me to it was a) RAID 1 would let me set up two 250gb pairs and have fault tolerant backup and b) the hot swappable ability to do just as you suggest in your post, buy cheap internal SATAs and make them the archiving system.


Same thng, NAS is just attached via a network rather than a direct connection. Commercial solutions often step up to the next level and use SAN (Storage Area Network) with very fast and expensive networking gear so that servers may share storage and allow easy expansion of storage.

With NAS things range from appliances (portable RAID units) to full blown systems.

Another option with NAS is to use an old computer with something like http://www.freenas.org/ (early and free version of a web based unix system admistration for NAS) and then max out the drives.

My issue with RAID 1, 3, 5, 6, 10, 0+1, 50, ... is that they still has single points of failure. They just has less single points of failure than a single drive.


Curious what others are doing. Anyone recommend a fast (firewire 800 or SATA), quiet, enclosure? Anyone have any data on failure rates of hard drives?

eSATA (http://www.sata-io.org/esata.asp)is an option. It is just sturdier connector, shielded cabling, and it is hot pluggable (no need to reboot). eSATA may be worth exploring if cost effective solutions exist. As this technology is still emerging, todays viewpoint may be very out of date in 6 months.

eSATA over a PCI based controller may have limited value unless every PCI slot is on its own bus (server style motherboards). A PCI-Express x4 eSATA controller would have serious appeal.

Firewire and USB are more likely to be available on others systems.

Firewire 800 (IEEE 1394b) is less common. I have never seen any reviews noting it helps with single drive. It may be more useful with chained drives.

Myself, I cannot countenance spending more than $500 USD per TB of storage.

I would also like to hear success stories for large 0.5+ TB of data storage.

some thoughts,

Sean

[Josh Liechty]

There are tons of enclosures on Newegg, but the reviews there border on useless, and I would also like to hear what's working for people who use external hard drives already.

Some cost data that I found while reading about Quantum's Govault system (http://www.quantum.com/Products/removabledrives/Index.aspx) (removable hard disk cartridges, like Iomega's Rev system) might be of interest. A 120GB Govault cartridge (excluding the drive) is US$2.16 per GB. 400GB external hard disks that come in an enclosure tend to be around $0.50/GB, with 250GB external drives costing approximately $0.44/GB. If one were to take a 300GB drive sans enclosure (~$99) and put it in an enclosure (~$30), it wouldn't be much different than a pre-built 250GB external drive, at $0.43/GB. This is just a sampling of a few options on Newegg, so it's quite possible that you could do something for less than what I've estimated here. These results do call into question, however, my thoughts that a removable disk cartridge system's advantages outweigh the cost penalty.

[Ray West]

There have been many removeable cartridge systems over the years, but they never seem to stay around more than a couple of years or so. I have used Panasonic md system, iomega and syquest. All of the drives gradually failed, some manufacturers went bust, and I have various cartridges that are of no use.

I think the best route at the moment is external hdds, put into your own usb cases. If you buy seagate bare drives, they have a five year warranty. If you buy a pre-assembled usb drive, containing a seagate drive, it carries the case/system guarantee, which will not be five years. Seagate do not offer a warranty to end users for their oem drives, at least not in the UK. Some external cases have cooling fans, and some keep the drive powered up all the time.

However, it is not that much more expensive to use any old pc on a network, loaded with internal drives, possibly a raid system, compared to a number of 'quality' external usb drive cases. SCSI drives are generally more reliable, but tend to be more expensive for the same capacity.

The more important the data is, the more copies you need to keep.

Best wishes

[Eric Perlberg]

Well at this point... I just purchased two 300gb D2 triple interface LaCie drives. I decided to do this instead of buying enclosures because I did some research on one enclosure company ICY BOX which everyone in the UK seems to carry. Some people were having real problems with getting their drives to show up, others weren't, so even though theory says things should be straight forward, practice shows that Murphy's Law and unanticipated gotchas operates here as well as everywhere else. For example, on a Mac at least, to have a bootable hard drive using Firewire 400, you need a recent chip set. The majors seem to all use appropriate ones but not all firewire 400 chipsets are compatible and enclosure manufacturers frequently don't say anything about functioning as startup disks on a Mac (small market for them I guess, except maybe Macally enclosures) . The issue of noise came up at another point. Without being able to verify the noise of either a fan (IcyBox doesn't use one) or the clutter of the drive mechanism (specific to the drive not the enclosure) and since I don't have time to sort through all kinds of non-photographic issues, I took the fast/easy way out.

I'm going to try to use the new 300gb drives using Macintosh's built in RAID software. I should be able to use RAID 1 thus eliminating (fingers crossed) the need to back up through software or manually.

I do think that long term archiving is an under discussed weakness of digital at this point though the future will undoubtably bring something more fit to purpose.

[Sean DeMerchant]

I'm going to try to use the new 300gb drives using Macintosh's built in RAID software. I should be able to use RAID 1 thus eliminating (fingers crossed) the need to back up through software or manually.


RAID 1 is still subject to singe points of failure:


A virus deleting one file will delete both copies.
A driver failure can corrupt both drives simultaneously.
User error deleting one file will delete both copies.
A lighting strike/power surge can wipe both out simultaneously.
...In short, a pair of external drives where one is always unplugged from the computer and the wall is much safer. But a single fire/earthquake/flood/hurricane/tornado/... can wipe out both drives.

All RAID 1 covers is hard drive failure.

enjoy,

Sean

[Rob Peterson]

My issue with RAID 1, 3, 5, 6, 10, 0+1, 50, ... is that they still has single points of failure. They just has less single points of failure than a single drive.

Seems to me that a RAID 5 or 10 protects against a drive failure, in my experience the most common failure. Yes, a controller failure could destroy the data, but such a failure seems quite rare.

Myself, I cannot countenance spending more than $500 USD per TB of storage.

I would also like to hear success stories for large 0.5+ TB of data storage.

Sean

Three years ago I installed in machines at two different locations RAID 5 arrays based on SCSI drives and Adaptec RAID controllers. Drives in both arrays failed, but I never lost access to the data. In one case the controller rebuilt the data on a hot spare before I could buy and install the replacement drive. Recovery from the failures involved shutting down the machine, replacing the failed drive, and restarting the machine. The controller rebuilt the failed drive's data while continuing to support data access.

Last year I added to one of the machines with a SCSI RAID an Adaptec RAID controller with 8 SATA ports, and connected 6 250GB drives. The 8 port controller solves the problem of how to connect more than 4 SATA drives, although case space becomes an issue. I spent just over $400 for the SATA RAID controller, and about $100 each for the drives. I now have 0.9TB of SATA RAID5 at $1,000/TB, but those are 12 month old prices. Today I'd build a 2TB (net) RAID 5 array for $420 + 8 300GB drives at $80/each, or $420 + $640 = $1,100/2TB, or start smaller with a higher per TB cost and incrementally add drives as I need more space. Note that adding another drive potentially expands the logical volume rather than becoming a new logical volume.

I consider my installation a success, and I intend to continue growing my SATA RAID solution.

My outstanding issue is upgrading my backup solution. DDS4 tape just doesn't work for TB storage! I'm considering a high capacity tape drive (LTO2 at 200GB/tape?) or a tape library.

Bob

[Jason Anderson]

RAID 1 is still subject to singe points of failure:
A virus deleting one file will delete both copies.
A driver failure can corrupt both drives simultaneously.
User error deleting one file will delete both copies.
A lighting strike/power surge can wipe both out simultaneously.
...In short, a pair of external drives where one is always unplugged from the computer and the wall is much safer. But a single fire/earthquake/flood/hurricane/tornado/... can wipe out both drives.

All RAID 1 covers is hard drive failure.

enjoy,

Sean
1. Fair point on the virus, but if you have the know how to set up a RAID array, you should also have the know how to implement both software and hardware firewalls, as well as the use of a reliable AV package.

2. A driver failure will cause a drive not to be recognized by the RAID card or the OS, so the drive really isn't corrupted, just inaccessible. Reinstalling the driver (often by simply rebooting) solves this problem.

3. If you delete a file accidentally, the backups that are not part of your active drives should address that problem. I won't go into a person adept enough to install RAID, but still inattentive enough to delete files...

4. The lightning argument will hold for ANY setup that does not have surge protection, so in this respect RAID 1 is no different than any other array, so I fail to see the point of this argument. The same point does hold that held in #'s 1 and 3 though - if you know enough to RAID, you should also know enough to include surge protection and UPS solutions strategies.

5. What else....? :)

An external drive relies on the person to actively make copies of everything. If that copy isn't made, there is no redundancy...RAID by definition, once set up, is redundant...Redundant Array of Independant Disks...

[Eric Perlberg]

One issue which has to be addressed by everyone contemplating serious backup is just how much is your data worth and where to draw the spending line. As I said in my inital post, data backup risk management is a slippery slope as one can continually come up with potential threats to one's data requiring ever more extreme solutions with ever increasing cost implications.

I don't know if its even possible to do but it might be a useful idea to develop some sort of heuristic which helps an individual decide where to draw the line. There are amateur photographers who face one sort of risk, professional fashion, wedding, glamour, news, etc. photographers who have to face another set of questions and then there is the professional artist working in photography who face yet a different set of issues.

[Andreas Kanon]

Right now i just mirror the data onto two different hard drives.
I am waiting for Blu Ray to start to ship in volume.
The specs i have seen on some blue ray readers/writers they can read regular cd and dvds just fine.
On one single Blu Ray disk you can get 25 or 50 Gb of data on a disk the same size as a DVD.
At least for me that will put a temporary end to my problems of backing up to a long term storage solution.

[Eric Perlberg]

I've been thinking that too with all the talk suddenly about Blue Ray. It will be interesting to see what the longevity of those disks prove to be.

[Andreas Kanon]

I've been thinking that too with all the talk suddenly about Blue Ray. It will be interesting to see what the longevity of those disks prove to be.

Agreed it will be interresting to see.
Personally i will be making 2 copies, one in the bank and one at home.

[Gary Ayala]

About a year or so ago, I found an old box filled with equally old photos. The snaps were more than two decades old and very well preserved (since they were never exposed to light). I've been scanning them and tossing them on my wedsite. Using that experience as a lesson ... I suggest printing your best stuff and boxing it. (Not very romantic, or quick ... but a solution nonetheless.)

My old snaps are here:

http://www.garyayala.smugmug.com/gallery/665619

[Nikolai Sklobovsky]

I was about to post a question, but decided to search first - and, man, was I rewarded by this thread...:-)

My HDD is almost filled up. When I got it less than 6 months ago I thought 250Gb will serve me 2-3 years... No such luck. RAW files use the space fast...;-)

DVDs do not seem to make the cut, since I'd have to burn one every week or so. Let alone you'd have to use a rather expensive media for the archiving purposes, since the el cheapo $.50 blanks won't last even 5 years.

External HDDs... Well, I have my doubts, too. Unless you keep them spinning they can deform. They can "the magnetic charge". Anyway, I have no reason to believe they will sit quietly in the closet for 5-10 years and then with a single power switch flip they will spring back to life..

Tape was kinda industry-standard for the backups for decades. Although I did see my good share of bad tapes to not trust them either. And they seem to be still quite expensive, albeit recently some of the devices became available for the low-end consumer market.

BluRay looks promising, but it's not here yet price-wise.

So I hope for the BluRay (or HDVD) to come in 2-3 years. The question is - how to survive until it's here.
My current line of thoughts - external 400-500GB HDD with compression turned on, which should make it into 600-800Gb. That would hopefully give me a piece of mind until the high-capacity optical writable media is here, and after that it may still serve well as a secondary storage device.

[Asher Kelman]

Blue Ray DVD burned with an inorganic chemical process is about to be available. This to me will make the best storage medium when the disks are stored vertical in in archival bags in a steel cabinet. The Blue RAy may turn out to be reliable for a decade or more.

Hard drives need to be examined for latent errors at least once a year and IMHO, backed up to new media every 5 years.

asher

[Sean DeMerchant]

Hard drives need to be examined for latent errors at least once a year and IMHO, backed up to new media every 5 years.


Unless there is a massive shift in the current evolution of HDD pricing, simply backing up to a new drive once a year (double the storage for the same price) would seem reasonable to me. My computer is currently maxed out* at ~1.6TB and I am within a few months or running out of storage. I am seriously considering buying 3.5" external firewire enclosures and 300 GB drives with the intent of buying larger drives as the sweet spot on the price curve changes.

But, using anything but HDD is hard for me to countainence as they are more reliable than all but tape media and the human cost in terms of time of burning CD/DVD/... is huge.

As to latent errors, there are two solutions.

1) Annual copies of pairs of drives to other pairs of drives plus running a disk check for bad sectors.

2) Parity files. Creating 100% parity data and then copying 1/2 the parity to each of a pair of disks increases storage usage by 33% while greatly increasing redundancy. Albeit, this too has great cost in terms of user time.

I personally use parity files for anything shifted to DVD+R (and once CDR which is prohibitively small now).

enjoy,

Sean


* Unless I replace drives with larger ones.

[Asher Kelman]

Hi Sean,

Love that flower and (is it a) fly!!

I'd get SATA, not more firewire as the Mac doesn't like too many for some reason and may give Kernel panics.

Now what do you use to check your disk media and how do you create your parity files? Did you ever say what OS you use?

Asher

[Sean DeMerchant]

Love that flower and (is it a) fly!!

I am assuming your are talking about the digger bee on the gum plant in the other thread. Thanks. :o) Or did I miss something.

Truth be told, I have shifted my shooting style to try and get additional shots for http://www.bugguide.net to get hlep on IDing the beautiful little ones and to add to their historical data.

I'd get SATA, not more firewire as the Mac doesn't like too many for some reason and may give Kernel panics.

My issue here is that I am looking at pairs of drive of which no more than one will ever be connnect to the system at once. If I go SATA, then it will be a $800 PCI-Express RAID 5/6 card and a stack of external exclosures do the lack of ports on hot swapable E-SATA cards. But truth be told, what I want is offline storage and Firewire and USB compatible (i.e., both and not either) drives are less expensive and can be disconnected from the power mains making it easy to store an archival copy of a full disk at a friend or family members (or a bank safe deposit box) to increase earthquake/flood/volcanic data survival.



Now what do you use to check your disk media and how do you create your parity files? Did you ever say what OS you use?


I would simply run checkdisk in XP as I run XP. And in practice, I would run checkdisk on the new and old disk, copy the files over, and then compare the files bit for bit to verify the copy.

I would like a Mac as I run bash as my shell anyway and I truly broke my teeth on computer systems using unix. But XP is cheaper and almost reliable. If I could make a business case for buyig a Mac Pro, I would. But I cannot. Hence, day to day at the command line I run bash shell under cygwin and ignore the MSFT nastiness under the hood.

But practically speaking, I have dual cores and 3 GB of RAM so performance deltas are minor until Adobe releases an x86 PS binary anyway

In short, I use XP, but I am OS agnostic so long as I can run my tools.

enjoy,

Sean (who is falling asleep at the keyboard)

[Brian Lowe]

As an computer network administrator I am always concerned about data storage and disaster recovery. So for storing my RAW files I use Jungle Disk it uses Amazon.com's S3™ Storage. Data is stored at multiple Amazon.com data centers around the country. Plus it very cheep .15 cents a gigabyte.

Living in Southern California we have earthquakes so, I like the idea of storing my photo data files in multiple data centers though out the country.

You may want to look into this I love having redundancy in data storage.



You have to ask yourself whats your photo data worth to you?


Brian

[Nikolai Sklobovsky]

Brian,

can you please provide a bit more details on this JungleDisk deal? Please??

TIA!

[Brian Lowe]

Hi Nik & others

Take a look for yourself and decide if it's for you.
All of the details are right here ==> http://www.jungledisk.com/

[Sean DeMerchant]

Hi Nik & others

Take a look for yourself and decide if it's for you.
All of the details are right here ==> http://www.jungledisk.com/

It is very expensive.

Consider:

initial upload of 250 GB -> $50.00
1 month of storage plus an upload of 25 GB -> $42.50
1 month of storage plus an upload of 25 GB -> $46.25
1 month of storage plus an upload of 25 GB -> $50.00
1 month of storage plus an upload of 25 GB -> $53.75
1 month of storage plus an upload of 25 GB -> $57.50
1 month of storage plus an upload of 25 GB -> $61.25
1 month of storage plus an upload of 25 GB -> $65.00
1 month of storage plus an upload of 25 GB -> $68.75
1 month of storage plus an upload of 25 GB -> $72.50
1 month of storage plus an upload of 25 GB -> $76.25
1 month of storage plus an upload of 25 GB -> $80.00
1 month of storage plus an upload of 25 GB -> $83.75

Yielding $807.50 USD for the first years of storage if you shoot a moderate 25 GB of new images per month. Over a period of years this will add up. And since I can get a Firewire/USB case and a 300GB drive for $175 USD with shipping I can get 600 GB of redundant (two copies) online storage for $700. Plus, the slower USB tranfer rate far exceeds my DSL upload speed meaning that the network backup is really slow. Add in that you are using beta software to achieve this storage and I cannot see the value proposition as periodically mailing a drive to a friend or relative would still be cheaper.

And if you shoot more than 25 GB a month this would be very expensive. Though if you can make a business case for it the distributed nature has disaster recovery value. But if your credit card expires the month the disaster hits and they simply delete your data as you cannot contact them to update payment info, you get hit again.

my $0.02,

Sean

[Alan T. Price]

In my opinion we are pretty well stuffed because there is no such thing as long term storage in this industry. 50 years is a pipe dream. 5 years is really stretching it. 2-3 years is likely. Tapes and discs (both optical and magnetic) all deteriorate. The only way to prevent data loss is to have a strict data management regime that will include documentation in case you should forget a step, and multiple on- and off-line and on- and off-site backups. But those individual back-up components need to be manged too.

Many of the posts to this thread have mentioned SATA drives. Well, consider this: what happened to PATA ? It's almost vanished in the space of a few months. SATA will go the same way one day. So will any media type. It's an inevitability of a commercially-oriented business world. The hardware and software manufacturers are looking after their interests; not mine, and not yours.

Even a well stocked SATA system with plenty of spare capacity will eventually need additional or replacement drives that will not fit or will not connect. Then you are up for another new system. Now that's a terrifying prospect because by then you will not necessarily be able to copy your data from the old system to the new system. All of those SATA drives in the dedicated SATA storage system will be rendered useless as soon as the dedicated system controller itself dies, and then you find that a new one cannot be found, and your SATA drives won't fit the shiny new PC you have by then.

Similarly, in a years time none of your PCI cards will find a new home. None of your 16-bit system software will find an operating system. You might think "so what?" but the software we have for some of the older drives including tape is 16 bit. Even some of the 32-bit software died when Windows XP went to SP2, but of course other new software won't work without SP2. My favourite automatic backup utility no longer works and is no longer supported since Symantec bought out PowerQuest. Thank you Symantec, for nothing. There are just too many examples of this for us to believe it won't happen again.

Even the comfort of having a warranty is lost when the manufacturer has gone bust - warranties no longer apply.

I don't know what the solution is but I'm inclined to think that it needs to be low-tech and cheap and to have only just enough capacity for the short term. It also needs to be off-line most of the time to prevent runaway software from attacking it when it attackes the rest of your PC.

Condider for example a pair of large-ish RAID 1 drives in a simple external controller so that...
(1) a separate PC and operating system is not required to drive it,
(2) if a drive dies you may be able to replace it,
(3) if the controller dies you may be able to replace that,
(4) if either the drive or the controller dies and cannot be replaced you can buy a whole new set of different technology or brand relatively cheaply
(5) when new technology comes out (such as SATA replacing PATA) you can buy a whole new set, with bigger capacity, before the old system has a chance to die,
(6) it is relatively affordable to replace it when required,
(7) with any luck it will still work when you upgrade to a new PC (and if it doesn't then there's time to replace it before you dispose of the old PC).

The longest-lasting image backups I've had to date are, believe it or not, on good old film. What does that say for the digital photography/computer industry ?

The bottom line, in my opinion, is that [I]It is important that the backup solution must not be allowed to outlive the technology or the computer/software support or the manufacturer or even the manufacture, or else you will eventually find your backup is compromised, and it may happen much sooner than later.

[Nikolai Sklobovsky]

Alan,

great post! Thank you very much!

Now I have a favor to ask: can you provide a sample configuration for your suggested solution? Something capable of storing, say, 1-2 Tb for the next 2-3 years?

Ta, mate!

[Alan T. Price]

Nik, I'm not sure how to help you out on that one. 2,000 GB is a lot of storage. What does it represent for you ? i.e. is it the total storage of backups, or the on-line storage, or the combined storage ? Is it for a single backup, or multiple backups ? Is a single backup in one device, or multiple devices ?


My own requirement is quite modest at this stage. Because of that I have not kept up with the latest available HDD holders / RAID controllers. Even if I had I would doubt that I would find most of them here in Perth. For example, the biggest HDDs at the popular store here is 500GB SATA2 (already superceding SATA). Their external cases are for single PATA or SATA drives. At a camera shop I found a 2-disc RAID box with LAN connection but only 10/100. So with what is readily available here I can't set up a stand-alone TB storage solution. I'd have to import one of the expensive options that obviously will have little or no local support.

Another shop has a Thecus Gigabit LAN 5xSATA2 RAID controller box with cpu and ram for $22, but that's gotta be a misprint. You can't buy anything for $22. Especially Australian dollars.

I'll enquire further when I get back to Perth, but in the meantime all can say is to repeat my advice to keep it simple and cheap so you afford to replace it early.

[Nikolai Sklobovsky]

Alan,
OK, got it, thanks!
I guess I need to do a bit of my own homework here;-)
Cheers!

[John_Nevill]

Have any of you guys tried the Buffalo Terastation. They are available upto 2TB with RAID 5. Its a cheap Linux SANS solution and provides both network and USB connectivty. I use them at work for moving masses of GIS/Telemetry data around between clients / suppliers.

For personal use, I use Lacie USB/Network drives and just add to them when needed. I keep one live and do differential back ups to the others. That way i'm not wearing out the backup drives through constant use.

I also keep DVD copies of everything.

[Alan T. Price]

none of those things are particularly cheap. You pay a hefty premium for what is basically a clever box to hold and manage the drives, and one of my concerns was what happens when that box dies - will Buffalo still be around ? Will the drives be readable in another box from a different company ? Do you just reformat and start again ? Do you maintain two of the boxes just in case one dies ?

The individual drive approach works too but has risks associated with it. Consider using them with Windows XP and formatting them with NTFS for maximum security and robustness. Now you have a choice of allocating a specific drive letter to each one so you can manage them better, but you will run out of letters and have to double-up eventually and that will cause grief for windows or you because no more than one drive partition can have a drive letter. Or you can not assign drive letters and let the system do it on the fly, in which case you risk losing track of your on-line drives.

It's still manageable but it takes more effort and awareness on your part.

DVDs are a good backup too (although Bluray will be more practical), but you need to create two sets in case one DVD goes wonky and loses data. It probably pays to duplicate it again every so many years just to improve longevity of the data. I use different brands for each copy in case one is worse than the other.

Differential backups are a risk in themselves. I once lost a whole set because something in the process trashed the master catalogue. I wasn't happy to lose not only the new backup but also the old one. Apart from that type of error, the master could be lost to defective media and then there may be difficulty extracting the rest of the data from the differential backups. It depends a lot on the software you use.

[Nill Toulme]

Something that should be factored into all this is that drives just keep getting cheaper. My data currently lives on a 1.1TB internal SATA RAID. It backs up nightly to a couple of external Seagate 400GB firewire drives (which I like a lot better than the chronically problematic Maxtors that preceded them).

Inspired partly by this thread and partly by general angst, I just buzzed over to Best Buy and picked up another Seagate external, this one 750GB, for $450. For the time being, that one will be used to back up the backups weekly, and it will live offline at my next door neighbor's house.

One of those nasty old Maxtors does still get used also. When I download a card to the PC, Downloader Pro automatically puts a second copy of the download over on that drive. I dump stuff off of it periodically as things make their way onto the other backups.

Nill
~~
www.toulme.net

[ChrisDauer]

There are tons of enclosures on Newegg, but the reviews there border on useless, and I would also like to hear what's working for people who use external hard drives already.

Some cost data that I found while reading about Quantum's Govault system (http://www.quantum.com/Products/removabledrives/Index.aspx) (removable hard disk cartridges, like Iomega's Rev system) might be of interest. A 120GB Govault cartridge (excluding the drive) is US$2.16 per GB. 400GB external hard disks that come in an enclosure tend to be around $0.50/GB, with 250GB external drives costing approximately $0.44/GB. If one were to take a 300GB drive sans enclosure (~$99) and put it in an enclosure (~$30), it wouldn't be much different than a pre-built 250GB external drive, at $0.43/GB. This is just a sampling of a few options on Newegg, so it's quite possible that you could do something for less than what I've estimated here. These results do call into question, however, my thoughts that a removable disk cartridge system's advantages outweigh the cost penalty.


I just picked up a 400 GB drive for $100 (+tax). I bought a very nice enclosure for $50 that does IDE and Sata. That brings me to $.375/GB. I should mention that the enclosure is nice because I can swamp new drives in when the current one gets full, and I don't have to invest in another enclosure. I'm not as worried about the drive degrading or falling apart when I put the next one in. I've been very fortunate in the past to let drives sit for 3-5 years and then pop them in and just have them work.

My current game plan is to move everything off the very old 160GB onto the new 400GB. Then, turn around and back up the important images on the 160 (second copy). Then remove all photos/backup photos off of all 3 of my internal system drives EXCEPT for current works in progress AND the best of the best which I send to friends now and again.

It has been very educational reading this thread. Thanks to all who contributed. I have been very curious as to what others do for there photo storage solutions. I know my solutions is extremely simple and not as involved as some of the more complex devices using the various RAID options.

[Asher Kelman]

Chris,

This is such an important step you have taken! congrats on keeping the cost down. Ideally you need to have one more such setup so you can backup the other drive each night.

Us the previous drive for off site storage of you key files or else as a scratch disk for Photoshop.

I'd love to know what make drive and box you ended up getting.

For one thing, I steer clear of getting LaCie big drives with more than one drive inside as that mayincrease the chance of trouble and the spanning of the drives may not be dealt with for you when trouble occurs after the short warranty period is over. I learned the hard way.

Asher

[Ray West]

I don't know if its been mentioned here, or elsewhere, but the 'buffalo linkstation' is a useful device. It sits on the network, has a reasonable sized inbuilt hdd, and also 2 usb 2 ports, one for a printer, one for another external hdd. It runs a cut down version of linux (which you never have to get into) and you can run it ith macs and windows. It also has automatic backup built in. Not expensive, takes up little power or space, and its quiet. They do bigger units too, but this system is simple.

Best wishes,

Ray

[Asher Kelman]

Hi Ray,

What connections does it have and are the drives removable? One things that is nice is when the box monitors for SMART fn of the drives.

I'm not sure I really understand how the link station differs from a regular hard drive or other simiar boxes.

I have moved away from firewire since it is problematic with a bunch of them causing G4 and G5 Macs to become unstable. It may be a combination of that with 3rd party memory.

Currently I buy 250 GB drives and have 5 removable and a bunch of Macsales.com (OWC) closed cases which work very well. each closed case has 2x500 GB and I have a spare one.

My filing is going slowly as I rename files according to year and move them to the new drives.

The big issue is finding some files are present with 5 copies and others just one with no backup.

Asher

[Ray West]

Hi Asher,

here's a link http://www.buffalotech.com/products/product-detail.php?productid=106&categoryid=22

As I said, it is a network device with print server, and a built in hdd, and the ability to have another external usb drive. It is not that much more expensive than a prebuilt external usb drive. As its a network device, it can copy files from its internal hdd, to its usb external one, or between other link stations. Some of the reviews on some other sites are out of date, some are wrong.

Buffalo do other, larger storage devices, too.

Best wishes,

Ray

[Asher Kelman]

Seems good, Ray. Just that the auto backup software is for PC users.

How would one utilize all the functions with a Mac? I guess one would need print server software? I know /mac has it for their own print servers. Who knows?

Asher

[Sean DeMerchant]

Seems good, Ray. Just that the auto backup software is for PC users.

How would one utilize all the functions with a Mac? I guess one would need print server software? I know /mac has it for their own print servers. Who knows?



Hi Asher,

This tech is old school. Way older than OS X or XP. And OS X derives from the Mach Unix kernel via a few generations. The files could be shared via Appleshare (may take a Linux extension), NFS (Network File System), or Samba (windows based network server protocol, a.k.a., SMB). On OS X you should be able to simply open a command line and type lpr (http://www.hmug.org/man/1/lpr.php) with the appropriate command line flags and print away. Albeit, this is old school (not really old school like punch cards) stuff and requires a command line. I cannot imagine Apple not supporting such in OS X. Last time I needed to network an OS X box (I think it was OS 10.2 IIRC) 10 minutes on the net told me where to look under finder to do it. Nonetheless, this technology was common (on large networks) in 1993 (5-7 generations ago in computing time).

Nonetheless, IEEE 1394 (Firewire) is faster than Fast Ethernet or wireless networks making such a solution low performing unless you have a Gigabit Ethernet LAN in the house. Add to that that you can unplug a Firewire or USB drive and have it disconnected from the mains/power very easily and you have a much better solution in discrete drives than by a network solution where data safety is your primary issue. An unplugged drive will be lost in a fire, tornado, earthquake, or hurricane, but lighting strikes will not hurt it nor will viruses when it is not connected to power. External hot-swappable (Firewire/USB/E-SATA) drives are the next best thing to offsite backups.

As to corrupt copies, that is easily corrected by validating copies via things like MD5/SHA1/CRC32/... hashes or direct bit for bit validation.

some thoughts but no answers,

Sean

[Tony Panzica]

This is a great thread and very informative. Of the 42 posts, there are 42 different solutions that everyone seems to have developed on their own. I would like to see a committee of say 5 of the heavier contributors develop some step by step solutions. Now you have to develop a different solution for an event shooter, a deadline conscience sports shooter, a wedding shooter, a nature shooter... Get the idea? Maybe at a convention, get a bunch of folks together to work. I dont know if this will work, but on paper it sounds cool. The problem that we have is as soon as you develop those solutions, the manufacturers change their product line up, pricing, availability.
My problem was reading how several of you bad mouthed the Maxtor drive. My brother, a top exec quality control guy who works for a chip manufacturer recommended it to me. I have the Maxtor III, 1TB drive, in fact I have 2 of them. Its fast and quite. I started out with film, then had the original 30D, used a Travan tape drive, have over 50GB on tape, then burned about 100 CDs, then burned 30 DVDs.
There are definite gaps in the Photo Industry for sure. But I wouldn't trade where I am at now compared to how I used to shoot with film. Can you imagine shooting a sporting event with only 36 shots on a roll? Then have to wait til you go to lab and hope you nailed it. You just have to stay on top of whats going on, then decide what fits your budget and workflow. Good luck...

[Cem_Usakligil]

Hi Tony,

You probably will have realised that a committee of 5 experts will never be able to come up with a single solution no matter how long they delibarate on this (LOL). You should digest the information given here and there and decide for yourself what works in your particular case and what does not.

Re. the Maxtor drives, this is how I see it:
They are one of the best (and biggest) brands out there and they produce good/affordable drives. Some of the products will inevitably be bad or wil have QC issuues. Some people will bad mouth the brand in general, some will praise them. Mind you, you can replace the brand name with Seagate, IBM/Hitachi, Samsung, whatever. Each and every one of them have people who either love or hate them. It is a bit like the silly Mac vs PC wars, really. Having used hundreds of PC hard disks from all major manufacturers in the past 25 years, I can only conclude that they all have been good on the average, even though many of them have crashed or sub-performed in that same period. What I'm trying to say is that one should ignore the negative feedback as long as it is not based on statistical and objective evidence. Maxtor drives are very good, and I'd certainly recommend one. Heck, I use many of those myself without problems so far. Take your pick, and use it. Don't forget to make BACKUPs though ;-).

Cheers,

Cem

[Dierk Haasis]

I would like to see a committee of say 5 of the heavier contributors develop some step by step solutions.

Design by committee - in the best of cases we get a camel [for those not knowing the expression: A camel is a horse designed by a committee]. IME - and I am heavily biased - committees are not very good since they look for the lowest common denominator (= compromise). That's a good thing when we are talking soft issues like how to organise schools and universities; it's decidedly bad if you need a workable result on hard issues.

Politicians, for instance, tend to look for a compromise even when talking about facts or mathematics. Two opposing parties come up with clearly opposite plans for taxing, one says we need a VAT that's 2 points higher, the other says they will not increase. After the election they have to come up with a compromise:

(0+2)/2

Curiously the committees came up with a surprising solution to this rather simple problem - 3 points increase.

My problem was reading how several of you bad mouthed the Maxtor drive.

I use a Maxtor One Touch II and a Maxtor One Touch III, and both seem to be very good.No problems, yet apart from a strange interaction over my Firewire connections [the III automatically disconnects when I switch on a Plextor DVD burner].

[Ray West]

In most circumstances, a camel performs far better than a horse.

I am coming to the conclusion, that in many ways it is best to use usb2 for everything you can. Cheap, fast enough for most things, portable, allows off site backup, easy to upgrade hdd's, etc. etc. Keyboard, track ball, hdds, cd/dvd r/w, network, etc. everything onto usb. As well as buffalo, linksys do a network usb hub. Also simple enough to use a cheap pc good enough for a file server.

Best wishes,
Ray

[Dierk Haasis]

In most circumstances, a camel performs far better than a horse.

Hence my 'in the best case'. The adage was obviously coined by some big city Victorian who was into looks and elegance more than into performance. Today it's often teh other way round, many folks go for over-powered, over-sized cross-country cars (SUVs and such nonsense), which are never ued off road, never driven remotely close to their limits. A camel is very good in deserts and near-deserts, a llama performs better in mountainous terrain, a horse is brillinat on relatively even savannah/prairie ground. It obviously looks better.

[Cem_Usakligil]

... It obviously looks better.
Now we are treading into your favourite territory Dierk, i.e. philosopy.
A horse obviously looks better to another horse, definitely!
Possibly not from a camel's perspective though ;-)

Cem
(LOL)

[Asher Kelman]

Hi Asher,

This tech is old school. Way older than OS X or XP. And OS X derives from the Mach Unix kernel via a few generations. The files could be shared via Appleshare (may take a Linux extension), NFS (Network File System), or Samba (windows based network server protocol, a.k.a., SMB). On OS X you should be able to simply open a command line and type lpr (http://www.hmug.org/man/1/lpr.php) with the appropriate command line flags and print away. Albeit, this is old school (not really old school like punch cards) stuff and requires a command line. I cannot imagine Apple not supporting such in OS X. Last time I needed to network an OS X box (I think it was OS 10.2 IIRC) 10 minutes on the net told me where to look under finder to do it. Nonetheless, this technology was common (on large networks) in 1993 (5-7 generations ago in computing time).

Nonetheless, IEEE 1394 (Firewire) is faster than Fast Ethernet or wireless networks making such a solution low performing unless you have a Gigabit Ethernet LAN in the house. Add to that that you can unplug a Firewire or USB drive and have it disconnected from the mains/power very easily and you have a much better solution in discrete drives than by a network solution where data safety is your primary issue. An unplugged drive will be lost in a fire, tornado, earthquake, or hurricane, but lighting strikes will not hurt it nor will viruses when it is not connected to power. External hot-swappable (Firewire/USB/E-SATA) drives are the next best thing to offsite backups.

As to corrupt copies, that is easily corrected by validating copies via things like MD5/SHA1/CRC32/... hashes or direct bit for bit validation.

some thoughts but no answers,

Sean

Sean,

I like these points:

1. We have already fast independant drives that become moderately virus-protected by unplugging.

2. Hot Swap SATA gives us "off-site" backup capability: and it's cheap.

3. Data is checkable. Here I'd like to have more detail.

What "out of the box" software is their available for Mac and Windows that can be added to our work flows? Is this already inherent in some of the smart boxes or file servers that won't break the bank?

Asher

[Sean DeMerchant]

3. Data is checkable. Here I'd like to have more detail.

What "out of the box" software is their available for Mac and Windows that can be added to our work flows? Is this already inherent in some of the smart boxes or file servers that won't break the bank?

For simple data verification on PCs one can use CDCheck:

http://www.kvipu.com/CDCheck/

This is about $50 for a commercial license and you can validated copied data easily before deleting it from your system and relying on your backup copies*. Such validation may take extra time, but it also leaves one a lot more comfortable deleting tens of GB of image files from a drive.

As to already included software there are likely md5, sha1, or direct difference (i.e., diff) executables on most *nix installs or they are available for free download. But they will require command line scripting. I do not know of any simple GUI based tools for OS X, but they likely exist.

enjoy,

Sean

* Note explicit plural.

[Eric Hiss]

Hi All,
I've been enjoying this thread and lots of points I had not thought about have been raised. I had been learning about this stuff the hard way - through 3 hard drive failures over the course of 6 months. I also know that DVD backups are not 100% :-(

Okay so what I have now. I run a highpoint 1.2Tb RAID which is connected via their pci-x card to my main working computer a Mac Quad G5. A fast connection is essential for a working volume. I am using Aperture to catalog and process most of my files from Canon 1Ds and 5D, but also use lightroom beta right now for my Leica DMR images until Apple can get organized to add compatibility for that. The main computer and working RAID are protected against power outages with a UPS. Everytime you loose power while the computer is operating you can corrupt a small portion of the disk. If that happens to be the header volume information...well you are screwed or are in for at least spent time in recovery. If your computer is not on a UPS then you should consider it. Laptops of course are protected. UPS also protects against lightning too.

Now my backup data is stored on a NAS server. I use the ReadyNAS NV with the same 4x400gb sata drives. This is much slower but I can let it run while I am away. I am backing up at irregular intervals right now because of the slow speed. The nice thing about having the back up on the internet is that you can provide remote access to data if needed and also you can locate the backup volume away from the main working volume to protect against loss to fire etc. What I am doing is to unplug the back up and take it to my studio so not really using the NAS features now. But the ReadyNAS has its own but smaller UPS.

It cost me a bundle for all this, but then I already know the cost of replacing lost images!
Eric

[Asher Kelman]

Could you explain this service Eric, how does it work and what's the cost?

Asher

[Paul Spatafora]

I finally ordered my 2Tb Buffalo Terastation. After much consideration, I decided to get this unit over all the other options mentioned in this thread. The reasons for this are:

1. I can run in a RAID 5 mode.
2. Buying the compatible hardware and assembling a RAID myself would have saved me a $100-$200 without any warranty or tested and proven software.
3. The box is ready to use and tested with easy to use software.
4. It has 4 USB2 ports and a print server port for networking.
5. DVD's or Dual Layer DVD's are a PIA and I'm not going to get sucked in the Beta vs. VHS fight that seems to be playing out between Blu Ray and HD. Besides point of entry is going to be very expensive. Consider that a DL DVD cost 4 times that of a single layer DVD and you can imagine the cost of the high capacity disks when they come out.
6. Single USB external drives provide no safety other than shutting them off.
7. HD's are still the cheapest, most backward compatible solution still today and within a RAID.

I think for now, a hot swappable system might be the safest of all the solutions above. And the management of the array looks simple. Even I can administer it, hopefully.

Great thread.

Paul

[Amy Goalen]

The topic of Digital Asset Management is such a big animal to get around in the digital age. A few months ago a read a wonderful book on this subject that address many of the concerns on this thread, "The DAM Book" by Peter Krough.

I can't say enough about this book and I also recommend it to all my students in my workshops. The one thing I have found is that Digital Asset Management is not cheap, but very necessary...which is why I finally took a deep breath and plunked down the business charge card.


Amy Goalen
Leaf/Phase One software trainer
Digital Technician

[Asher Kelman]

Hi Amy,

For me at least, the DAM Book is something that should be in everyone's list to reread and rethink one's current storage system. It's just storage 101!

I BTW, don't follow Peter's advice fully, but I like his overall ideas.

The main thing is to have a standard system and to have upgradeable to any new storage medium and software system.

I do not run huge iview media Pro catalogs, since they get klutsy can crash. I am generous in having multiple catalogs by month organization and type of work or even camera.

So I have scores of specific catalogs, for example: an M8 catalog, a 2006_12_14_Smashbox catalog and say a 2006_Art Master catalog.

There is absolutely no functional worth for me to have my art projects in the same catalog as a wedding or gala social.

Now maybe I'll get closer to Peter's methodology on my next rereading.

I'm still working on my organization of files from before I started a logical DAM system.

BTW, OWC.com is selling a Terrabyte (2x500 GB Seagate SATA) external kit in a beautiful black aluminum case wit a very quiet fan, 4 about $430.

It goes with Macs but I cants see why it wouldn't work with PC's too.

Asher