The timing of the outage couldn’t have been much worse for east coast Slack users, as reports began spiking just before 7 AM EST when many were logging on for the day. Within minutes, AWS’ status page was updated to confirm that the company had detected a power outage “within a single data center within a single Availability Zone (USE1-AZ4) in the US-EAST-1 Region.” Restoration efforts got underway quickly, and AWS reported progress on restoring power within 18 minutes of its initial confirmation. However, the restoration process remained ongoing for several more hours until AWS noted that power had fully been restored at 9:51 AM EST. Even then, the company’s support page warned that some network connectivity issues continued to linger for a portion of impacted EC2 instances. It also revealed that some customers of its EBS storage service were impacted by “degraded IO performance” during the outage. Despite the apparent pattern of outages cropping up during December, the three incidents have very little in common. The first on December 7 also impacted the US-EAST-1 Region, but it related to an automated network capacity scaling issue rather than a power outage. The second downtime happened all the way across the country in the US-WEST-1 and WEST-2 regions and was related to a network connectivity issue. Whether there is an underlying reason for the sudden spike in AWS downtimes, or the company has just had a run of extremely bad luck remains to be seen. However, the apps, services, games, and websites that have come to rely on AWS for their own stability are almost certainly beginning to take a long, hard look at the impact these outages are having on their own bottom lines.
