In Perspective Planet: Backup and Cloud Computing for Mission Critical Tasks: Example - AWS!

[Asher Kelman]

I booked a place for the keynote speakers at the AWS Amazon cloud this morning. What amazed me is the rapidity with with cloud computing has subsumed and reinvented the ways major companies provide services at lower capital cost and advances without time delays with scalabilities that are almost unimaginable.

Asher Kelman: Screen snapshot November 28th 2012
AWS Keynote Speaker Andy Jassy, Senior VP AWS Amazon Web Services

Most remarkably one can consider the US Presidential Election of 2012 where 18 months ago there were essentially no full time employees to the height of the campaign where two competing industries, each with up to millions of employees and volunteers across all 52 states had to do battle. The Romney campaign built their own servers and went to work well funded and with very capable management. The Obama campaign turned their back on that paradigm, The instead went to the cloud. That meant no risks of server downtime, leveraging Amazons scalability and being able to focus on their own innovation. So what did they invent? Well they developed over 200 custom cloud based applications all integrated with one database but each dedicated to one task. So there were modules for assessing, nurturing and following through with donors, modules to analyze the habits and leanings of each separate US voter so they knew where to send their phone calls and volunteers.

They did not enter the cloud on their own. Obama's team did something exceptionally smart. They entered via SAS services. SAS is a company founded and run by Sanja J. Poonen and is expert in all business and enterprise level giant real time transactions, done to exacting security and within legal oversight boundaries. So they followed in the steps of the 2012 Olympics, Nasdaq, Dancing with the Stars and the journeys of the Mars rover, Curiosity and entered the encrypted cloud world of high level computing.

So how does this effect us? Well everything we do will be in the cloud. Who knows, some among us might be already contributing to its capability and others no doubt with invent software that will make their mark. I just thought it would be fun to contemplate a world where an election campaign was able to carry out 500,000 transactions per second securely, obey all election regulations, have a record of what as done for transparency and allow people to compete for votes in an open environment. This is a powerful change to democracy. It's a challenge to us to keep it honest!

Asher

Oh, BTW, for those who didn't know, Governor Romney's election campaign servers broke down and President Obama won a second term as President of the United States! I have a distinct feeling that before long, some republicans are going to sit down with software engineers and map out a cloud strategy of their own!

Read more on how democratic political partiers will have to learn to wage war here.

[Asher Kelman]

So, how are you using the cloud for your photography or work applications?

BTW, if you use Smugmug, then you already are!

Asher

[Asher Kelman]

I said Governor Romney's servers broke down, but what really happened is instructive.

Aurich Lawson / Thinkstock: Photo

It was supposed to be a "killer app," but a system deployed to volunteers by Mitt Romney's presidential campaign may have done more harm to Romney's chances on Election Day—largely because of a failure to follow basic best practices for IT projects.

Called "Orca," the effort was supposed to give the Romney campaign its own analytics on what was happening at polling places and to help the campaign direct get-out-the-vote efforts in the key battleground states of Ohio, Florida, Pennsylvania, Iowa, and Colorado.

Instead, volunteers couldn't get the system to work from the field in many states—in some cases because they had been given the wrong login information. The system crashed repeatedly. At one point, the network connection to the Romney campaign's headquarters went down because Internet provider Comcast reportedly thought the traffic was caused by a denial of service attack.

As one Orca user described it to Ars, the entire episode was a "huge clusterf***." Here's how it happened.

Read more of the meltdown in Ars Technica

BTW, isn't the picture of the whale funny given the name of the software was "Orca"!

[Asher Kelman]

Reduced Redundancy Storage.

If one is working with colleagues on large image files, say for an exhibition and you want a number of sites to have easy internet access then Reduced Redundancy Storage may be an economical choice. It should be that you have the data already stored and that is tis recreatable. The data is pretty damn safe, but only to a level of 99.99%. So that in one year, you could have a 0.01$ chance of losing data. Regularly stored data on AWS is 99.9999999999% safe per year.

Reduced Redundancy Storage (RRS) is an Amazon S3 storage option that enables customers to reduce their costs by storing noncritical, reproducible data at lower levels of redundancy than Amazon S3’s standard storage. It provides a cost-effective, highly available solution for distributing or sharing content that is durably stored elsewhere, or for storing thumbnails, transcoded media, or other processed data that can be easily reproduced. The RRS option stores objects on multiple devices across multiple facilities, providing 400 times the durability of a typical disk drive, but does not replicate objects as many times as standard Amazon S3 storage.

Reduced Redundancy Storage is:

• Backed with the Amazon S3 Service Level Agreement for availability.
  
• Designed to provide 99.99% durability and 99.99% availability of objects over a given year. This durability level corresponds to an average annual expected loss of 0.01% of objects.
  
• Designed to sustain the loss of data in a single facility.

Is anyone using AWS for their pictures. If you don't know, then if you use Shutterbug, then you are already a user! I am just thinking about it since it appears flexible and backed by one of the strongest companies. what cloud services do folk here use and why?

Asher

[James Lemon]

Originally Posted by Asher Kelman

Reduced Redundancy Storage.

If one is working with colleagues on large image files, say for an exhibition and you want a number of sites to have easy internet access then Reduced Redundancy Storage may be an economical choice. It should be that you have the data already stored and that is tis recreatable. The data is pretty damn safe, but only to a level of 99.99%. So that in one year, you could have a 0.01$ chance of losing data. Regularly stored data on AWS is 99.9999999999% safe per year.

Reduced Redundancy Storage (RRS) is an Amazon S3 storage option that enables customers to reduce their costs by storing noncritical, reproducible data at lower levels of redundancy than Amazon S3’s standard storage. It provides a cost-effective, highly available solution for distributing or sharing content that is durably stored elsewhere, or for storing thumbnails, transcoded media, or other processed data that can be easily reproduced. The RRS option stores objects on multiple devices across multiple facilities, providing 400 times the durability of a typical disk drive, but does not replicate objects as many times as standard Amazon S3 storage.

Reduced Redundancy Storage is:

• Backed with the Amazon S3 Service Level Agreement for availability.
  
• Designed to provide 99.99% durability and 99.99% availability of objects over a given year. This durability level corresponds to an average annual expected loss of 0.01% of objects.
  
• Designed to sustain the loss of data in a single facility.

Is anyone using AWS for their pictures. If you don't know, then if you use Shutterbug, then you are already a user! I am just thinking about it since it appears flexible and backed by one of the strongest companies. what cloud services do folk here use and why?

Asher

I have my own cloud system on my own in house network. It consists of multiple security layers. I have a 4 terabyte disc that is mirrored to another disc in case of disc failure plus a recycle bin in case something gets deleted accidently. I then manually backup this volume 1 to another volume 2 set up the same way.

In total 16 terabytes with 4 terabytes of space used for my picture archive and it is easy to expand these capabilities.

So I don't rent space and I am not concerned about having any of my clouds disappearing into thin air. Using somebody else's cloud system is like going to a financial adviser and having them say don't worry about risking your life savings (we'll do it for you).

James

[Asher Kelman]

Originally Posted by James Lemon

I have my own cloud system on my own in house network. It consists of multiple security layers. I have a 4 terabyte disc that is mirrored to another disc in case of disc failure plus a recycle bin in case something gets deleted accidently. I then manually backup this volume 1 to another volume 2 set up the same way.

In total 16 terabytes with 4 terabytes of space used for my picture archive and it is easy to expand these capabilities.

So I don't rent space and I am not concerned about having any of my clouds disappearing into thin air. Using somebody else's cloud system is like going to a financial adviser and having them say don't worry about risking your life savings (we'll do it for you).

James

James,

I was at Henry's in San Francisco and a sale an recounted the sad story of the fire that destroyed his server and 25 years of photography with all his redundant backups! That is why one needs off site storage in addition to any local measures.

To lose all your fm data on AWS it would require that 5 or more servers in diverse locations as North America, South America, Japan, Taiwan, Australia, Israel, South Africa, France, the U.K., Germany, Austria, Denmark, Sweden and more would all be destroyed pretty well simultaneously!

That is awefully unlikely!

Another approach is to back up to a mirrored raid array in an alternative location, with a friend or relative. That's inexpensive, but a wise and proactive precaution!

Asher

[James Lemon]

Originally Posted by Asher Kelman

James,

I was at Henry's in San Francisco and a sale an recounted the sad story of the fire that destroyed his server and 25 years of photography with all his redundant backups! That is why one needs off site storage in addition to any local measures.

To lose all your fm data on AWS it would require that 5 or more servers in diverse locations as North America, South America, Japan, Taiwan, Australia, Israel, South Africa, France, the U.K., Germany, Austria, Denmark, Sweden and more would all be destroyed pretty well simultaneously!

That is awefully unlikely!

Another approach is to back up to a mirrored raid array in an alternative location, with a friend or relative. That's inexpensive, but a wise and proactive precaution!

Asher

I said multiple security layers it is very easy to wire into a fire proof safe. Witch I have done fire is not a problem in any event. But companies do go broke a other things can happen as well . My point is that they are not held accountable if they loose your stuff. There are many issues that I can think of that are nothing but liabilities when you rely on others. Did you want to rent some space from me or just borrow it for awhile? lol

James

[Asher Kelman]

My way is local redundancy, offsite family backup and then the cloud!

Everyone has to find there own comfort level!

Your way with a fireproof safe is fascinating. Do you have wires into the safe or just put updated drives I there every so often?

Asher

[James Lemon]

Originally Posted by Asher Kelman

My way is local redundancy, offsite family backup and then the cloud!

Everyone has to find there own comfort level!

Your way with a fireproof safe is fascinating. Do you have wires into the safe or just put updated drives I there every so often?

Asher

You could do either or I have a Freind that updates hard drives and keeps them in a Saftey deposit box at a local bank. I have everything on my network.

James

[Tom dinning]

Fail safe is never fail safe.

In the year 2000 all of my photographs and negatives were destroyed in a fire. The fire was as a result of an irate ex-wife with a box of matches seeking revenge. My only regret was that I was not present at the time so I might have at least warmed my hands or barbecued a shrimp or two.

These days I have two copies of my files and spend as much time as possible securing my relationship with Christine; a much more satisfying diversion from the anguish brought about by byte security or cloud collaterals.

In addition, the experience of losing that which i considered valuable and irreplaceable, quite emphatically pointed out how wrong and misled I was.

There's nothing quite so tacky a a photographer who considers his photos of more value than the person who looks at them.

[Asher Kelman]

Originally Posted by Tom dinning

Fail safe is never fail safe.

In the year 2000 all of my photographs and negatives were destroyed in a fire. The fire was as a result of an irate ex-wife with a box of matches seeking revenge. My only regret was that I was not present at the time so I might have at least warmed my hands or barbecued a shrimp or two.

These days I have two copies of my files and spend as much time as possible securing my relationship with Christine; a much more satisfying diversion from the anguish brought about by byte security or cloud collaterals.

In addition, the experience of losing that which i considered valuable and irreplaceable, quite emphatically pointed out how wrong and misled I was.

There's nothing quite so tacky a a photographer who considers his photos of more value than the person who looks at them.

Tom,

Of course, having good terms with the wife is top of the list! Mine has the keys to everything.

Fire has ruined so many photographer's collections. This is a tragedy that can be prevented. As to backup, having one copy at a mate's home or in a bank vault or anywhere else is a pretty good safeguard. For pictures that cannot be replaced and one values highly for whatever personal or business reasons, one should consider storage on a cloud server.

Today the prices are so low.

Asher

[James Lemon]

Photographers are sleep walking into photographic Armageddon warn experts (see link below)

http://www.bravenewcamera.com/opblog...r-photographer

[Asher Kelman]

Originally Posted by James Lemon

Photographers are sleep walking into photographic Armageddon warn experts (see link below)

http://www.bravenewcamera.com/opblog...r-photographer

A very major point, James. All digital storage fails gradually. Error repair can heal damaged records. However, having actual prints in several locations obviates the absolute necessity for translating a digital record to an actual image.

Asher

[Nicolas Claris]

Originally Posted by Asher Kelman

Another approach is to back up to a mirrored raid array in an alternative location, with a friend or relative. That's inexpensive, but a wise and proactive precaution!

Asher

All my back-ups are mirrored and placed into a safe at the bank.
Low cost and well guarded!

[Doug Kerr]

This is to summarize the backup practice here, which is described in more detail here:

http://www.openphotographyforums.com...ad.php?t=20416

On each of our two computers, an external drive (in a dock at that computer) is kept synchronized with the internal operating drive (not in real time, but automatically brought up-to-date every night). This duplicate drive is bootable, and should the internal drive fail, the duplicate can be quickly mounted in its place and the machine will be back in operation.

In addition, every night, all data files on both computers (including all photo files, both as they came in from the camera and as processed from various "shoots") are automatically backed up to another external drive on my computer (in a separate dock). This backup system uses "versioning": up to three copies (of progressive "age") can be kept of every file (with unique variations of the filename so they can coexist on the backup drive).

There are two of each of the three external hard drives in this system (the "duplicate" drive on each computer and the "backup drive", on my computer). One set is kept in a bank safety deposit box. The two sets are swapped about monthly.

No clouds (yes, we have some lovely ones here every couple of weeks) have been disturbed in the making of this system.

Best regards,

Doug

[James Lemon]

Originally Posted by Doug Kerr

This is to summarize the backup practice here, which is described in more detail here:

http://www.openphotographyforums.com...ad.php?t=20416

On each of our two computers, an external drive (in a dock at that computer) is kept synchronized with the internal operating drive (not in real time, but automatically brought up-to-date every night). This duplicate drive is bootable, and should the internal drive fail, the duplicate can be quickly mounted in its place and the machine will be back in operation.

In addition, every night, all data files on both computers (including all photo files, both as they came in from the camera and as processed from various "shoots") are automatically backed up to another external drive on my computer (in a separate dock). This backup system uses "versioning": up to three copies (of progressive "age") can be kept of every file (with unique variations of the filename so they can coexist on the backup drive).

There are two of each of the three external hard drives in this system (the "duplicate" drive on each computer and the "backup drive", on my computer). One set is kept in a bank safety deposit box. The two sets are swapped about monthly.

No clouds (yes, we have some lovely ones here every couple of weeks) have been disturbed in the making of this system.

Best regards,

Doug

Looks like you have things well under control Doug.

James

[Doug Kerr]

Hi, James,

Originally Posted by James Lemon

Looks like you have things well under control Doug.

Well, I hope so. We never know what safeguards will work under whatever disasters actually visit us!

Thanks.

Best regards,

Doug