Open Photography Forums  
HOME FORUMS NEWS FAQ SEARCH

Go Back   Open Photography Forums > OPF Welcome Hall > Breaking News

Breaking News Updates, innovations, equipment: moderated!

Reply
 
Thread Tools Display Modes
  #1  
Old November 28th, 2012, 10:34 AM
Asher Kelman Asher Kelman is offline
OPF Owner/Editor-in-Chief
 
Join Date: Apr 2006
Posts: 31,769
Default Backup and Cloud Computing for Mission Critical Tasks: Example - AWS!

I booked a place for the keynote speakers at the AWS Amazon cloud this morning. What amazed me is the rapidity with with cloud computing has subsumed and reinvented the ways major companies provide services at lower capital cost and advances without time delays with scalabilities that are almost unimaginable.




Asher Kelman: Screen snapshot November 28th 2012
AWS Keynote Speaker Andy Jassy, Senior VP AWS Amazon Web Services


Most remarkably one can consider the US Presidential Election of 2012 where 18 months ago there were essentially no full time employees to the height of the campaign where two competing industries, each with up to millions of employees and volunteers across all 52 states had to do battle. The Romney campaign built their own servers and went to work well funded and with very capable management. The Obama campaign turned their back on that paradigm, The instead went to the cloud. That meant no risks of server downtime, leveraging Amazons scalability and being able to focus on their own innovation. So what did they invent? Well they developed over 200 custom cloud based applications all integrated with one database but each dedicated to one task. So there were modules for assessing, nurturing and following through with donors, modules to analyze the habits and leanings of each separate US voter so they knew where to send their phone calls and volunteers.

They did not enter the cloud on their own. Obama's team did something exceptionally smart. They entered via SAS services. SAS is a company founded and run by Sanja J. Poonen and is expert in all business and enterprise level giant real time transactions, done to exacting security and within legal oversight boundaries. So they followed in the steps of the 2012 Olympics, Nasdaq, Dancing with the Stars and the journeys of the Mars rover, Curiosity and entered the encrypted cloud world of high level computing.

So how does this effect us? Well everything we do will be in the cloud. Who knows, some among us might be already contributing to its capability and others no doubt with invent software that will make their mark. I just thought it would be fun to contemplate a world where an election campaign was able to carry out 500,000 transactions per second securely, obey all election regulations, have a record of what as done for transparency and allow people to compete for votes in an open environment. This is a powerful change to democracy. It's a challenge to us to keep it honest!

Asher

Oh, BTW, for those who didn't know, Governor Romney's election campaign servers broke down and President Obama won a second term as President of the United States! I have a distinct feeling that before long, some republicans are going to sit down with software engineers and map out a cloud strategy of their own!

Read more on how democratic political partiers will have to learn to wage war here.
__________________
Follow us on Twitter at @opfweb

Our purpose is getting to an impressive photograph. So we encourage browsing and then feedback. Consider a link to your galleries annotated, C&C welcomed. Images posted within OPF are assumed to be for Comment & Critique, unless otherwise designated.

Last edited by Asher Kelman; March 28th, 2016 at 08:41 AM.
Reply With Quote
  #2  
Old November 28th, 2012, 11:00 AM
Asher Kelman Asher Kelman is offline
OPF Owner/Editor-in-Chief
 
Join Date: Apr 2006
Posts: 31,769
Default

So, how are you using the cloud for your photography or work applications?

BTW, if you use Smugmug, then you already are!

Asher
__________________
Follow us on Twitter at @opfweb

Our purpose is getting to an impressive photograph. So we encourage browsing and then feedback. Consider a link to your galleries annotated, C&C welcomed. Images posted within OPF are assumed to be for Comment & Critique, unless otherwise designated.
Reply With Quote
  #3  
Old November 28th, 2012, 11:12 AM
Asher Kelman Asher Kelman is offline
OPF Owner/Editor-in-Chief
 
Join Date: Apr 2006
Posts: 31,769
Default

I said Governor Romney's servers broke down, but what really happened is instructive.




Aurich Lawson / Thinkstock: Photo

It was supposed to be a "killer app," but a system deployed to volunteers by Mitt Romney's presidential campaign may have done more harm to Romney's chances on Election Day—largely because of a failure to follow basic best practices for IT projects.

Called "Orca," the effort was supposed to give the Romney campaign its own analytics on what was happening at polling places and to help the campaign direct get-out-the-vote efforts in the key battleground states of Ohio, Florida, Pennsylvania, Iowa, and Colorado.

Instead, volunteers couldn't get the system to work from the field in many states—in some cases because they had been given the wrong login information. The system crashed repeatedly. At one point, the network connection to the Romney campaign's headquarters went down because Internet provider Comcast reportedly thought the traffic was caused by a denial of service attack.

As one Orca user described it to Ars, the entire episode was a "huge clusterf***." Here's how it happened.


Read more of the meltdown in Ars Technica

BTW, isn't the picture of the whale funny given the name of the software was "Orca"!
__________________
Follow us on Twitter at @opfweb

Our purpose is getting to an impressive photograph. So we encourage browsing and then feedback. Consider a link to your galleries annotated, C&C welcomed. Images posted within OPF are assumed to be for Comment & Critique, unless otherwise designated.
Reply With Quote
  #4  
Old March 27th, 2016, 01:45 PM
Asher Kelman Asher Kelman is offline
OPF Owner/Editor-in-Chief
 
Join Date: Apr 2006
Posts: 31,769
Default

Reduced Redundancy Storage.

If one is working with colleagues on large image files, say for an exhibition and you want a number of sites to have easy internet access then Reduced Redundancy Storage may be an economical choice. It should be that you have the data already stored and that is tis recreatable. The data is pretty damn safe, but only to a level of 99.99%. So that in one year, you could have a 0.01$ chance of losing data. Regularly stored data on AWS is 99.9999999999% safe per year.


Reduced Redundancy Storage (RRS) is an Amazon S3 storage option that enables customers to reduce their costs by storing noncritical, reproducible data at lower levels of redundancy than Amazon S3’s standard storage. It provides a cost-effective, highly available solution for distributing or sharing content that is durably stored elsewhere, or for storing thumbnails, transcoded media, or other processed data that can be easily reproduced. The RRS option stores objects on multiple devices across multiple facilities, providing 400 times the durability of a typical disk drive, but does not replicate objects as many times as standard Amazon S3 storage.

Reduced Redundancy Storage is:
  • Backed with the Amazon S3 Service Level Agreement for availability.
  • Designed to provide 99.99% durability and 99.99% availability of objects over a given year. This durability level corresponds to an average annual expected loss of 0.01% of objects.
  • Designed to sustain the loss of data in a single facility.

Is anyone using AWS for their pictures. If you don't know, then if you use Shutterbug, then you are already a user! I am just thinking about it since it appears flexible and backed by one of the strongest companies. what cloud services do folk here use and why?

Asher
__________________
Follow us on Twitter at @opfweb

Our purpose is getting to an impressive photograph. So we encourage browsing and then feedback. Consider a link to your galleries annotated, C&C welcomed. Images posted within OPF are assumed to be for Comment & Critique, unless otherwise designated.
Reply With Quote
  #5  
Old March 27th, 2016, 05:22 PM
James Lemon James Lemon is offline
Senior Member
 
Join Date: May 2013
Location: Canada
Posts: 1,378
Default

Quote:
Originally Posted by Asher Kelman View Post
Reduced Redundancy Storage.

If one is working with colleagues on large image files, say for an exhibition and you want a number of sites to have easy internet access then Reduced Redundancy Storage may be an economical choice. It should be that you have the data already stored and that is tis recreatable. The data is pretty damn safe, but only to a level of 99.99%. So that in one year, you could have a 0.01$ chance of losing data. Regularly stored data on AWS is 99.9999999999% safe per year.


Reduced Redundancy Storage (RRS) is an Amazon S3 storage option that enables customers to reduce their costs by storing noncritical, reproducible data at lower levels of redundancy than Amazon S3’s standard storage. It provides a cost-effective, highly available solution for distributing or sharing content that is durably stored elsewhere, or for storing thumbnails, transcoded media, or other processed data that can be easily reproduced. The RRS option stores objects on multiple devices across multiple facilities, providing 400 times the durability of a typical disk drive, but does not replicate objects as many times as standard Amazon S3 storage.

Reduced Redundancy Storage is:
  • Backed with the Amazon S3 Service Level Agreement for availability.
  • Designed to provide 99.99% durability and 99.99% availability of objects over a given year. This durability level corresponds to an average annual expected loss of 0.01% of objects.
  • Designed to sustain the loss of data in a single facility.

Is anyone using AWS for their pictures. If you don't know, then if you use Shutterbug, then you are already a user! I am just thinking about it since it appears flexible and backed by one of the strongest companies. what cloud services do folk here use and why?

Asher
I have my own cloud system on my own in house network. It consists of multiple security layers. I have a 4 terabyte disc that is mirrored to another disc in case of disc failure plus a recycle bin in case something gets deleted accidently. I then manually backup this volume 1 to another volume 2 set up the same way.

In total 16 terabytes with 4 terabytes of space used for my picture archive and it is easy to expand these capabilities.

So I don't rent space and I am not concerned about having any of my clouds disappearing into thin air. Using somebody else's cloud system is like going to a financial adviser and having them say don't worry about risking your life savings (we'll do it for you).

James
Reply With Quote
  #6  
Old March 27th, 2016, 06:03 PM
Asher Kelman Asher Kelman is offline
OPF Owner/Editor-in-Chief
 
Join Date: Apr 2006
Posts: 31,769
Default

Quote:
Originally Posted by James Lemon View Post
I have my own cloud system on my own in house network. It consists of multiple security layers. I have a 4 terabyte disc that is mirrored to another disc in case of disc failure plus a recycle bin in case something gets deleted accidently. I then manually backup this volume 1 to another volume 2 set up the same way.

In total 16 terabytes with 4 terabytes of space used for my picture archive and it is easy to expand these capabilities.

So I don't rent space and I am not concerned about having any of my clouds disappearing into thin air. Using somebody else's cloud system is like going to a financial adviser and having them say don't worry about risking your life savings (we'll do it for you).

James

James,

I was at Henry's in San Francisco and a sale an recounted the sad story of the fire that destroyed his server and 25 years of photography with all his redundant backups! That is why one needs off site storage in addition to any local measures.

To lose all your fm data on AWS it would require that 5 or more servers in diverse locations as North America, South America, Japan, Taiwan, Australia, Israel, South Africa, France, the U.K., Germany, Austria, Denmark, Sweden and more would all be destroyed pretty well simultaneously!

That is awefully unlikely!

Another approach is to back up to a mirrored raid array in an alternative location, with a friend or relative. That's inexpensive, but a wise and proactive precaution!

Asher
__________________
Follow us on Twitter at @opfweb

Our purpose is getting to an impressive photograph. So we encourage browsing and then feedback. Consider a link to your galleries annotated, C&C welcomed. Images posted within OPF are assumed to be for Comment & Critique, unless otherwise designated.
Reply With Quote
  #7  
Old March 27th, 2016, 06:20 PM
James Lemon James Lemon is offline
Senior Member
 
Join Date: May 2013
Location: Canada
Posts: 1,378
Default

Quote:
Originally Posted by Asher Kelman View Post
James,

I was at Henry's in San Francisco and a sale an recounted the sad story of the fire that destroyed his server and 25 years of photography with all his redundant backups! That is why one needs off site storage in addition to any local measures.

To lose all your fm data on AWS it would require that 5 or more servers in diverse locations as North America, South America, Japan, Taiwan, Australia, Israel, South Africa, France, the U.K., Germany, Austria, Denmark, Sweden and more would all be destroyed pretty well simultaneously!

That is awefully unlikely!

Another approach is to back up to a mirrored raid array in an alternative location, with a friend or relative. That's inexpensive, but a wise and proactive precaution!

Asher
I said multiple security layers it is very easy to wire into a fire proof safe. Witch I have done fire is not a problem in any event. But companies do go broke a other things can happen as well . My point is that they are not held accountable if they loose your stuff. There are many issues that I can think of that are nothing but liabilities when you rely on others. Did you want to rent some space from me or just borrow it for awhile? lol

James
Reply With Quote
  #8  
Old March 27th, 2016, 07:01 PM
Asher Kelman Asher Kelman is offline
OPF Owner/Editor-in-Chief
 
Join Date: Apr 2006
Posts: 31,769
Default

My way is local redundancy, offsite family backup and then the cloud!

Everyone has to find there own comfort level!

Your way with a fireproof safe is fascinating. Do you have wires into the safe or just put updated drives I there every so often?

Asher
__________________
Follow us on Twitter at @opfweb

Our purpose is getting to an impressive photograph. So we encourage browsing and then feedback. Consider a link to your galleries annotated, C&C welcomed. Images posted within OPF are assumed to be for Comment & Critique, unless otherwise designated.
Reply With Quote
  #9  
Old March 27th, 2016, 07:10 PM
James Lemon James Lemon is offline
Senior Member
 
Join Date: May 2013
Location: Canada
Posts: 1,378
Default

Quote:
Originally Posted by Asher Kelman View Post
My way is local redundancy, offsite family backup and then the cloud!

Everyone has to find there own comfort level!

Your way with a fireproof safe is fascinating. Do you have wires into the safe or just put updated drives I there every so often?

Asher
You could do either or I have a Freind that updates hard drives and keeps them in a Saftey deposit box at a local bank. I have everything on my network.

James
Reply With Quote
  #10  
Old March 27th, 2016, 11:52 PM
Tom dinning Tom dinning is offline
Senior Member
 
Join Date: Mar 2012
Location: Darwin NT Australia
Posts: 2,032
Default

Fail safe is never fail safe.

In the year 2000 all of my photographs and negatives were destroyed in a fire. The fire was as a result of an irate ex-wife with a box of matches seeking revenge. My only regret was that I was not present at the time so I might have at least warmed my hands or barbecued a shrimp or two.

These days I have two copies of my files and spend as much time as possible securing my relationship with Christine; a much more satisfying diversion from the anguish brought about by byte security or cloud collaterals.

In addition, the experience of losing that which i considered valuable and irreplaceable, quite emphatically pointed out how wrong and misled I was.

There's nothing quite so tacky a a photographer who considers his photos of more value than the person who looks at them.
__________________
I'm working on it, OK!

http://notesfromthecamera.blogspot.com.au/
Reply With Quote
  #11  
Old March 28th, 2016, 10:05 AM
Asher Kelman Asher Kelman is offline
OPF Owner/Editor-in-Chief
 
Join Date: Apr 2006
Posts: 31,769
Default

Quote:
Originally Posted by Tom dinning View Post
Fail safe is never fail safe.

In the year 2000 all of my photographs and negatives were destroyed in a fire. The fire was as a result of an irate ex-wife with a box of matches seeking revenge. My only regret was that I was not present at the time so I might have at least warmed my hands or barbecued a shrimp or two.

These days I have two copies of my files and spend as much time as possible securing my relationship with Christine; a much more satisfying diversion from the anguish brought about by byte security or cloud collaterals.

In addition, the experience of losing that which i considered valuable and irreplaceable, quite emphatically pointed out how wrong and misled I was.

There's nothing quite so tacky a a photographer who considers his photos of more value than the person who looks at them.
Tom,

Of course, having good terms with the wife is top of the list! Mine has the keys to everything.

Fire has ruined so many photographer's collections. This is a tragedy that can be prevented. As to backup, having one copy at a mate's home or in a bank vault or anywhere else is a pretty good safeguard. For pictures that cannot be replaced and one values highly for whatever personal or business reasons, one should consider storage on a cloud server.

Today the prices are so low.

Asher
__________________
Follow us on Twitter at @opfweb

Our purpose is getting to an impressive photograph. So we encourage browsing and then feedback. Consider a link to your galleries annotated, C&C welcomed. Images posted within OPF are assumed to be for Comment & Critique, unless otherwise designated.
Reply With Quote
  #12  
Old March 28th, 2016, 11:04 AM
James Lemon James Lemon is offline
Senior Member
 
Join Date: May 2013
Location: Canada
Posts: 1,378
Default

Photographers are sleep walking into photographic Armageddon warn experts (see link below)

http://www.bravenewcamera.com/opblog...r-photographer
Reply With Quote
  #13  
Old March 28th, 2016, 11:10 AM
Asher Kelman Asher Kelman is offline
OPF Owner/Editor-in-Chief
 
Join Date: Apr 2006
Posts: 31,769
Default

Quote:
Originally Posted by James Lemon View Post
Photographers are sleep walking into photographic Armageddon warn experts (see link below)

http://www.bravenewcamera.com/opblog...r-photographer

A very major point, James. All digital storage fails gradually. Error repair can heal damaged records. However, having actual prints in several locations obviates the absolute necessity for translating a digital record to an actual image.

Asher
__________________
Follow us on Twitter at @opfweb

Our purpose is getting to an impressive photograph. So we encourage browsing and then feedback. Consider a link to your galleries annotated, C&C welcomed. Images posted within OPF are assumed to be for Comment & Critique, unless otherwise designated.
Reply With Quote
  #14  
Old March 28th, 2016, 01:14 PM
Nicolas Claris Nicolas Claris is offline
Administrator/Moderator
 
Join Date: Apr 2006
Posts: 5,272
Default

Quote:
Originally Posted by Asher Kelman View Post
Another approach is to back up to a mirrored raid array in an alternative location, with a friend or relative. That's inexpensive, but a wise and proactive precaution!

Asher
All my back-ups are mirrored and placed into a safe at the bank.
Low cost and well guarded!
Reply With Quote
  #15  
Old March 29th, 2016, 07:01 AM
Doug Kerr Doug Kerr is offline
Senior Member
 
Join Date: May 2006
Location: Alamogordo, New Mexico, USA
Posts: 8,042
Default

This is to summarize the backup practice here, which is described in more detail here:

http://www.openphotographyforums.com...ad.php?t=20416

On each of our two computers, an external drive (in a dock at that computer) is kept synchronized with the internal operating drive (not in real time, but automatically brought up-to-date every night). This duplicate drive is bootable, and should the internal drive fail, the duplicate can be quickly mounted in its place and the machine will be back in operation.

In addition, every night, all data files on both computers (including all photo files, both as they came in from the camera and as processed from various "shoots") are automatically backed up to another external drive on my computer (in a separate dock). This backup system uses "versioning": up to three copies (of progressive "age") can be kept of every file (with unique variations of the filename so they can coexist on the backup drive).

There are two of each of the three external hard drives in this system (the "duplicate" drive on each computer and the "backup drive", on my computer). One set is kept in a bank safety deposit box. The two sets are swapped about monthly.

No clouds (yes, we have some lovely ones here every couple of weeks) have been disturbed in the making of this system.

Best regards,

Doug
Reply With Quote
  #16  
Old March 29th, 2016, 10:48 AM
James Lemon James Lemon is offline
Senior Member
 
Join Date: May 2013
Location: Canada
Posts: 1,378
Default

Quote:
Originally Posted by Doug Kerr View Post
This is to summarize the backup practice here, which is described in more detail here:

http://www.openphotographyforums.com...ad.php?t=20416

On each of our two computers, an external drive (in a dock at that computer) is kept synchronized with the internal operating drive (not in real time, but automatically brought up-to-date every night). This duplicate drive is bootable, and should the internal drive fail, the duplicate can be quickly mounted in its place and the machine will be back in operation.

In addition, every night, all data files on both computers (including all photo files, both as they came in from the camera and as processed from various "shoots") are automatically backed up to another external drive on my computer (in a separate dock). This backup system uses "versioning": up to three copies (of progressive "age") can be kept of every file (with unique variations of the filename so they can coexist on the backup drive).

There are two of each of the three external hard drives in this system (the "duplicate" drive on each computer and the "backup drive", on my computer). One set is kept in a bank safety deposit box. The two sets are swapped about monthly.

No clouds (yes, we have some lovely ones here every couple of weeks) have been disturbed in the making of this system.

Best regards,

Doug
Looks like you have things well under control Doug.

James
Reply With Quote
  #17  
Old March 29th, 2016, 11:42 AM
Doug Kerr Doug Kerr is offline
Senior Member
 
Join Date: May 2006
Location: Alamogordo, New Mexico, USA
Posts: 8,042
Default

Hi, James,

Quote:
Originally Posted by James Lemon View Post
Looks like you have things well under control Doug.
Well, I hope so. We never know what safeguards will work under whatever disasters actually visit us!

Thanks.

Best regards,

Doug
Reply With Quote
Reply

Bookmarks

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
A Parable for Our Times? Murray Foote Landscape - Travel 38 June 24th, 2012 10:30 PM
In Perspective Planet: Apple and Google's world! Adobe and Microsoft lose a little clout! Asher Kelman Layback Cafe 9 May 5th, 2010 09:00 AM
News: Free Amazing Orchestra Performance Saturday April 24th, Santa Monica, California Asher Kelman Layback Cafe 3 April 27th, 2010 12:16 AM
It's a small world James Newman Pet's Corner: Professional as well as enjoyable photos of your pets. 3 June 16th, 2008 07:07 PM
The (DMF) world as we know it.. leonardobarreto.com Medium Format & Large Format Cameras 14 November 6th, 2007 08:42 AM


All times are GMT -7. The time now is 07:56 AM.


Posting images or text grants license to OPF, yet © of such remain with its creator. Still, all assembled discussion © 2006-2014 Asher Kelman (all rights reserved) Posts with new theme or unusual image might be moved/copied to a new thread!