Open Photography Forums  
HOME FORUMS NEWS FAQ SEARCH

Go Back   Open Photography Forums > Digital Darkroom > Digital Asset Management: access & storage of image files.

Reply
 
Thread Tools Display Modes
  #1  
Old January 16th, 2007, 01:52 AM
Sue Butler Sue Butler is offline
New Member
 
Join Date: Jun 2006
Location: Adelaide, South Australia
Posts: 21
Default Upload storage

Hi All,

I'm just getting into all the Digital Asset Management stuff and am reading Peter Krogh's book at the moment. Excellent!
And I'm starting to sort through all my files and get some sort of chronological and sensible order into them.

My husband told me about this site http://carbonite.com/
What do you think about using online storage?
Does it seem to be a sensible option or not?

Currently I have about 600G of RAW, tiff and jpg files spread over 3-4 hard drives so need to do something about storage rather soon!
My husband isn't a fan of putting files onto DVD so I have to look at other options.

Looking forward to some interesting replies.

Thank you
Sue

PS. This is my first post here but I've done a lot of reading of other messages here. It's a great site and I've learnt a lot already.
Please forgive my log in name 'Susie'. I've sent a PM to Michael to change it but I haven't had a reply as yet.
Reply With Quote
  #2  
Old January 16th, 2007, 02:17 AM
John Beardsworth John Beardsworth is offline
Member
 
Join Date: May 2006
Location: London
Posts: 57
Default

Hi Sue

I haven't heard of carbonite, but so much depends on the speed of transmission and the time you have to spend babysitting the operation and verifying it. Peter discussed it recently and shares my view that you're probably better off doing offsite backups yourself (unless you're on the road a lot).

The key is not to have all your eggs in one basket. So back up to DVD - gold type and on more than one copy. That's where what Peter calls buckets is so helpful. If your hard drive crashes, restoring it is reassuringly low tech, just whacking in one sequentially numbered DVD after another. And back up to another hard drive - these are rapidly becoming bigger and cheaper. You may find it better to get your husband to store a backup drive at work.

John
Reply With Quote
  #3  
Old January 16th, 2007, 05:36 AM
Nill Toulme Nill Toulme is offline
Senior Member
 
Join Date: Jun 2006
Location: Atlanta, Georgia USA
Posts: 1,407
Default

I don't even want to have to contemplate restoring half a TB or more of data from DVDs or the internet. External HDs are plentiful, cheap and reasonably reliable. My half a TB of data lives on a 1.1TB internal RAID5 in my graphics workstation. It backs up, automatically and nightly, to two 400GB Seagate external firewire drives. Those get backed up in turn, manually and weekly or so, to another 750GB Seagate external that lives in the meantime over at my neighbor's house.

Seagate just announced 1TB drives, so the 750 giggers will be moving into the pricing sweet spot.

Nill
~~
www.toulme.net
Reply With Quote
  #4  
Old January 16th, 2007, 05:45 AM
Ray West Ray West is offline
Senior Member
 
Join Date: Jun 2006
Location: UK - Somerset
Posts: 1,703
Default

Hi Susie,

Have you looked at the carbonite figures?

You have 600Gb of images - they are saying it is only ok for 50gb or so http://www.carbonite.com/faqs.aspx?#how_much_storage

even if 600Gb ok, then it will take you 300days to upload, and 60days to restore, using their software.

Is that what you want?

If your 600Gb of images is worth something, then get a few hdd raid systems, and have done with it. Perhaps go through your files, delete or convert to jpegs if you can, to reduce the storage requirements. Be hard on yourself. Home written dvd's will not last.

Best wishes,

Ray
Reply With Quote
  #5  
Old January 16th, 2007, 06:30 AM
Marian Howell Marian Howell is offline
Member
 
Join Date: May 2006
Location: massachusetts south coast
Posts: 225
Default

hi Susie!
welcome to OPF! i'm in agreement with Nill on this. i don't want to think about how tedious the restoration would be from dvds :) not to mention keeping a good database of what is on them! i'm too lazy! i got a hot swap bay and multiple hard drives (currently 300gb), each with its own enclosure tray. the current working drive is backed up to duplicate hard drives daily. one set is kept offsite. and with the hot swap bay i can access the local drives readily when needed. not the most sophisticated setup, but i feel comfortable with it. and as Nill says, with the prices of hard drives coming down all the time, the drives get bigger all the time. i've been with this system for several years, and i occasionally plug in each of the drives to verify them, especially the older ones. i imagine that this year i'll be getting some larger drives and will combine some of the older drives on them as well, as a third backup!
__________________
marian
Reply With Quote
  #6  
Old January 16th, 2007, 07:19 AM
John Beardsworth John Beardsworth is offline
Member
 
Join Date: May 2006
Location: London
Posts: 57
Default

Obviously you can never have too much hard drive space and I mentioned taking one offsite, but note that the original poster refers to having 3-4 hard drives for this 600Gb - that makes a DVD based approach more workable, if tedious.

John
Reply With Quote
  #7  
Old January 16th, 2007, 07:25 AM
Asher Kelman Asher Kelman is offline
OPF Owner/Editor-in-Chief
 
Join Date: Apr 2006
Posts: 33,884
Default

Hot swap drives need to be rugged. I've found variable performance of the Seritiek 4 bay SATA enclosure. Sometimes one has to resinsert several times to get the connection to the backpane.

I switched to independant OWC dual enclosures, SATA, 2x500 GB and am happy with 5 of them!

My lines of firewire drives sit unused. Multiple firewire drives in series caused kernel panic crashes, the dreaded black screen of death that descends slowly with an overlay of code.

The current 5GB DVD's are slow and useless for retrospective backup but great for immediate use. The the Blue Ray

"LaCie Blu-ray Disc Burner

For those not hip to the latest technology, Blu-ray (developed by Sony) is a high-density disc format that stores much more than a standard CD and plays back media in High Definition. And leave it to our neighbor, LaCie to produce a compact, USB 2.0/FireWire burner. This great little burner is fast, compact, quiet and allows you to easily store up to 50GB of data on just one disc.
Models will ship at the end of this month and are priced at $1149.00. Please call to pre-order."http://www.powermax.com/



Asher
Reply With Quote
  #8  
Old January 16th, 2007, 07:44 AM
Tim Smith Tim Smith is offline
Member
 
Join Date: Nov 2006
Location: Central Maine, U.S.A.
Posts: 39
Default

I'll add another testimonial for a RAID enclosure and enough drive storage capacity to allow you to grow. I have a Wiebetech Silver SATA set up as a RAID 1 (mirrored) with a third tray/drive that I take offsite to the safe deposit box. Built like iron, quiet, and the simplest backup for me so far. My clients have required me to sign an agreement that I will maintain a rock-solid back up system. The mirrored RAID seems like the perfect solution... a belt AND suspenders.
Reply With Quote
  #9  
Old January 16th, 2007, 08:08 AM
Nill Toulme Nill Toulme is offline
Senior Member
 
Join Date: Jun 2006
Location: Atlanta, Georgia USA
Posts: 1,407
Default

Tim I'm not sure I agree with that. As I understand it, a RAID1 mirror only protects you against drive failure, not against all the other things that can jump up and bite you, especially the greatest threat of all in my case operator headspace errors. If you write a bad file to, or delete a good file from, your RAID1, you do it on both sides of the mirror. Then you're down to your backup(s).

Am I missing something?

Nill
~~
www.toulme.net
Reply With Quote
  #10  
Old January 16th, 2007, 02:12 PM
Tim Smith Tim Smith is offline
Member
 
Join Date: Nov 2006
Location: Central Maine, U.S.A.
Posts: 39
Default

I'm not an expert by any means on RAID. But here's how I use it. The first thing I do at the start of a new backup period (every 6 months) is to completely copy my entire internal drive to the fresh RAID, allowing them to become startup disks. Then, my daily work on client projects is run on my internal 300 GB drive. At the end of each day, I backup the contents of just my client folder to the RAID using a software called Super Duper. It checks to see which files have newer creation dates (meaning I've modified them in some way or are brand new) and copies only those changed files to the RAID backup. Should there be any significant change to either system software or application software, I'll back up the entire disk.

The RAID then mirrors or completely duplicates that first drive to a second one. I now have 2 independent copies of everything on my hard drive. Which essentially means 3 copies if you include the HD. Pretty good insurance against drive failure. Once a week, I replace one of the trays with the third RAID tray/drive and once it's completely copied the data on the RAID, use it to take to the bank for off-site storage.

Which provides lots of redundancy and security. You may be asking "what about if I want to go back to a version of a file that existed yesterday?" Maybe I made an error in an edit and want to return to a previous state. I can do that each day until I do the actual backup. But really it comes down to a working methodology. I never open a file and make edits without copying the file to a new version first. Never.

So again, this is what I've been doing, which may or not be the right way for everyone. I'm certainly open to suggestions for a better system.

P.S. I never throw files away either. Honest!
Reply With Quote
  #11  
Old January 16th, 2007, 02:45 PM
Sue Butler Sue Butler is offline
New Member
 
Join Date: Jun 2006
Location: Adelaide, South Australia
Posts: 21
Default Thank you !!

Hi Everyone !!

What a great bunch of replies first thing this morning. I don't know about you all but here in Adelaide, South Australia it was 40 C yesterday and it's a welcome relief this morning to have cloud cover and a temp of about 30 C!
Great information and very helpful.
I can certainly understand that putting everything onto DVD's would be a long, tedious process and I'm not completely happy about uploading my files to an online site. Besides, like Ray mentioned it is only 50G at a time and yes, I do know how long it would take to upload 600G to the carbonite site!!
It seems that lots of hard drives and backing up to hard drives seems to be one of the best and popular ways of archiving files.

Thank you all very much.
Sue
Reply With Quote
  #12  
Old January 16th, 2007, 04:57 PM
Sean DeMerchant Sean DeMerchant is offline
Inactive
 
Join Date: Jun 2006
Location: Whidbey Island, Washington State, USA
Posts: 735
Default Backup Is Not Version Control

Quote:
Originally Posted by Tim Smith
You may be asking "what about if I want to go back to a version of a file that existed yesterday?" Maybe I made an error in an edit and want to return to a previous state.
Backups are a very poor choice of tools for version control. I would suggest using Version Cue if you want it simple and integrated with Adobe products. Or, if you want a serious tool check out Subversion which is free and a rather nice tool. There are several inexpensive version control products out there too (Perforce comes to mind). Or if you want the creme de la creme then I would suggest looking at ClearCase (large learning curve, but it is the one others aspire to be.

If you really want good backups, then I would suggest using a version control repository and backing that up rather than just the files. This can be very helpful with detail oriented clients who want the version from 3 weeks ago Tuesday again. Albeit, version control is not very useful if you fail to give descriptive comments on checking in files.

Currently, I can go back to any version of most of the projects I work on.

Albeit, using version control on image files can consume a lot of disk space. But using backups for version control is somewhat like using a metal coat hanger to fix a broken tail pipe rather than a welding torch as over time it will fail.

some thoughts,

Sean
__________________
[I]"In the End, we will remember not the words of our enemies, but the silence of our friends." [/I]- Martin Luther King Jr. (1929-1968)
[URL="http://www.envisagement.com/"]http://www.envisagement.com/[/URL]
Reply With Quote
  #13  
Old January 16th, 2007, 07:10 PM
Tim Smith Tim Smith is offline
Member
 
Join Date: Nov 2006
Location: Central Maine, U.S.A.
Posts: 39
Default

I made a feeble attempt to use Version Cue with the first Adobe Suite. It wasn't pretty. I've heard it has since improved but have been either too lazy or too busy to get back to trying it. It might be worthwhile to check it out again. Thanks for the suggestion!
Reply With Quote
  #14  
Old January 16th, 2007, 09:56 PM
Asher Kelman Asher Kelman is offline
OPF Owner/Editor-in-Chief
 
Join Date: Apr 2006
Posts: 33,884
Default

Quote:
Originally Posted by Sean DeMerchant
Albeit, using version control on image files can consume a lot of disk space.
Not if one uses Aperture! The version space is trivial since it is only a small set of instructions not the full images files that have to be saved as a version.

Asher
Reply With Quote
  #15  
Old January 17th, 2007, 03:52 AM
Cem_Usakligil Cem_Usakligil is offline
Senior Member
 
Join Date: Jun 2006
Posts: 5,939
Default

Quote:
Originally Posted by Sean DeMerchant
Backups are a very poor choice of tools for version control. ....
Hi Sean,

I understand precisely what you're saying and agree with it. On top it, I'd like to make a few personal observations if I may :-).

There are a few aspects of backing up data that some of us might get confused with so I'd like to clarify how I see this.

1) Backups for covering any contingencies such as hard disk crashes (version 1).
This issue might be resolved by setting up a RAID 1/5/6/10 array. Many "novice" people tend to think that having a mirrored RAID1 array is enough for backup purposes. Well, it is not. It only helps surviving a disk crash, nothing more, nothing less. RAID0 is in any case useless for any backup purposes since it does not offer any contingency, but speed improvement only.
In this situation, if you do something stupid like deleting an essential file or if a virus destroys your data, RAID does offer no protection at all. This is just a short term insurance against harware failure.

2) Backups for covering any contingencies such as hard disk crashes (version 2).
One can typically duplicate (ie copy) the same file by copying it onto different hard disks (internal or extarnal) or onto DVDs. This method offers a similar level of contingency to a RAID array, only the recovery process is slower. The advantage WRT version 1 is that it is immune to operator errors and/or virusses (expect in the case of on-line secondary disk drives in your PC that are also prone to virus attacks).

3) Backups for making sure that you can have access to your files in case your house burns down or your computer (including the RAID array) gets stolen, etc.
Similar to version 2, this can be achieved in many ways. Use external hard disks as backup medium and store them physically at a different location. Or use DVDs and store them off-site. Whatever.

4) Backups for restoring your system (OS) if it crashes.
Firstly, one shoud always use a seperate hard disk or partition to keep one's data files. This way, one can backup and restore the OS independent of any data. One can use various backup tools to backup the OS: Ghost, Drive Image, Partition Copy, MS Backup, etc.
This is an essential activity and must be conducted every time something essential changes on your OS such as installing new (major) programs for the first time.
The backups must be preferably kept on seperate, external hard disks or DVDs. Best option is to keep them off-site for maximum security.

5) Backsups for achieving version control.
One might want to keep various versions of a certain file while it evolves. In that case, just backing up daily/weekly is not a solution, as Sean has pointed out. He has given a few pointers as to how one can achieve real version control. There is also much good advice to be found in publications/sites such as the DAM book. Everything you do within the version control is not backing up data. Even if one does version control, one should still do the above mentioned steps 1-3.

<begins personal rambling>
So now I'd like to address my pet dislike. I don't understand why so many folks out there are so obsessed with RAID arrays. Sure, if you are running primary applications on your system which your business is dependent on, a RAID array is an absolute necessity in order to minimise the downtimes in case the of hard disk crashes. For anybody else, especially for people like myself who are doing this as a hobby, a RAID array can be counterproductive. Let me mention a few (not so obvious) disadvantages:
- all the disks spin at the same time, so the power supply must be able to handle higher peak currents (can cause stability issues)
- all the disks spin at the same time, so more noise is generated and more heat is to be dissipated
- all the disks spin at the same time, so per disk operational running time is much higher than in case of individual disks, which will statistically lead to more often/earlier disk crashes
- Oh, did I mention more expensive (i.e. less available disk space)?
The advantages are obviously:
- less downtime due to disk crashes
- better read/write performance and data bandwidth

For me, the advantages of RAID are not important enough to use one. What I do instead is that I use many individual on-line and external hard disks accross which my data are distributed and duplicated. Besides, modern SATA drives achieve a throughput of about 60 MB, which is more than enough to handle any workload I can throw at them (such as non-linear DV editing, HD editing, RAW processing, etc).

So, now I've got it off my chest ;-)
</ends personal rambling>

Cheers,

Cem
Reply With Quote
  #16  
Old January 17th, 2007, 05:11 AM
Tim Smith Tim Smith is offline
Member
 
Join Date: Nov 2006
Location: Central Maine, U.S.A.
Posts: 39
Default

I was quite content using external drives as backups before Sarbanes Oxley. The redundancy required by my paying clients makes a RAID setup a viable alternative. And to be honest, now that I have it, I find it extremely easy to use and a great comfort. So much of this is situation- and user-preference-dependent. What works best for one might be overkill for another.
Reply With Quote
Reply

Bookmarks

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are Off
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -7. The time now is 10:15 AM.


Posting images or text grants license to OPF, yet of such remain with its creator. Still, all assembled discussion 2006-2017 Asher Kelman (all rights reserved) Posts with new theme or unusual image might be moved/copied to a new thread!