Storm breaks Amazon cloud services - againMore trouble in the East Coast datacenter as storms destroy power connections and services for Netflix, Pinterest and more.
Article by Penny JonesElectrical storms have plagued Amazon's Web Services
Amazon Web Services in the US-East-1 region suffered another outage late last week - less than two weeks after announcing a cable fault in a power distribution system of its electricity provider.
This time, however, wild electrical storms over Virginia were to blame, and these affected more than just users of Netflix, Pinterest and Instagram.
An indication of the scale of the outage - storms brought down power connections for more than one million customers of Dominion Virginia Power on the evening of June 29. At 8:21pm, the AWS health dashboard for the region posted: “We are investigating connectivity issues for a number of instances in the US-EAST-1 Region”. Ten minutes later: “We are investigating elevated errors rates for APIs in the US-EAST-1 (Northern Virginia) region, as well as connectivity issues to instances in a single availability zone.” And then at 8:40pm: “We can confirm that a large number of instances in a single Availability Zone have lost power due to electrical storms in the area. We are actively working to restore power.”
Power was restored at 8:49pm but it appears recovery was still taking place for some time. At 10:36 PM Amazon wrote: “We continue to bring impacted instances and volumes back online. As a result of the power outage, some EBS volumes may have inconsistent data.”
On the Help section of its site, popular online photo sharing site Instagram said the electrical storm had affected most of its services, and it was working through the night to restore services.
Many customers in the region were still without power on July 1, according to a New York Times report. So even if Amazon services were up, those using devices that needed plugging in would not have missed the services placed in Amazon’s cloud.
Dominion Power said it would have most power services restored by Tuesday night and the more severely damaged areas would not regain power until next Sunday. Dominion said unlike hurricanes, the provider was unable to predict the severity of the storm before it occurred. “Unlike a hurricane, this storm could not be forecasted well ahead of time by the National Weather Service," Rodney Blevins, VP of Electric Distribution Operations at Dominion said."That is complicating restoration efforts because crews and supplies could not be positioned in advance."
It is not the first time Amazon has suffered because of an electrical storm. In August 2011 a utility supplying power to Amazon’s Dublin datacenter was also affected by lightning.
In this instance back-up generators kicked in after the lightning strike took down a 10MW transformer.
At the time, AWS said it was likely that the datacenter’s programmable logical controllers, which synchronize electrical phases between generators, were to blame.
More internet woes due to time bug
Meanwhile, a number of other websites experienced problems over the weekend as official time keepers endeavoured to hold clocks back by a second to keep them in sync with the earth’s daily spin.
According to a report by Wired, Reddit, Mozilla, Gawker and many other websites that did not prepare for the change experienced technical issues as a result of the Leap Second bug.
This was not a new occurrence, time keepers have had to play God in Greenwich since first introducing atomic clocks in the early 1970s.
According to Australian newspaper The Courier Mail, airline Qantas’ computer systems crashed between 11am and 11:48am after the global booking system it uses – Amadeus Altea – was affected by the Bug.
This then brought down Qantas’ own website as customers raced to alert the airline of problems with their reservations.