The internet is broken — much of it that is. It really isn’t your imagination.
Me: “Alexa, turn on the living room lights.”
Alexa: “Living room lights is not responding.”
In the darkness I made my way to each individually controlled smart plug and light to manually turn them on. I got about half of them on with the bonus of a cut on my hand from reaching behind a piece of furniture.
My experience was only the tip of the Internet failure iceberg on Tuesday as Amazon’s Web Server farm business and the thousands of companies relying on AWS began to experience disruptions to downright complete outages.
Amazon’s AWS is the world’s largest server and data services provider to governments, companies and universities which are experiencing disruptions.
Were any of these apps misbehaving recently?
- Alexa skills controlling smart home devices failing
- McDonalds app
- PayPal
- Shopify
- Venmo
- NetFlix
- Disney+
- Prime Video
- Xfinity
- Fanduel
- iRobot vacuums won’t connect to the app that drives them
Grocery Store Glitches
I could not use my Whole Food apps to check out at a neighborhood market with the cashier saying that all of Amazon’s systems were having problems at the store.
Amazon package routing and deliveries delayed
On Reddit some Amazon warehouse and delivery employees were reporting that their Amazon company apps were broken preventing the ability to scan and route packages.
Why is this happening and what can we do?
As of this afternoon, the only thing Amazon posted for hours about the titanic-sized trouble came in a statement saying the it was “actively workin towards recover”. Hope my doctor never says those words to a family member.
Later in the afternoon, Amazon’s’ AWS Service Health Dashboard’ gave more detailed insight into the colossal ongoing disruption:
[11:26 AM PST] We are seeing impact to multiple AWS APIs in the US-EAST-1 Region. This issue is also affecting some of our monitoring and incident response tooling, which is delaying our ability to provide updates. Services impacted include: EC2, Connect, DynamoDB, Glue, Athena, Timestream, and Chime and other AWS Services in US-EAST-1. The root cause of this issue is an impairment of several network devices in the US-EAST-1 Region. We are pursuing multiple mitigation paths in parallel, and have seen some signs of recovery, but we do not have an ETA for full recovery at this time. Root logins for consoles in all AWS regions are affected by this issue, however customers can login to consoles other than US-EAST-1 by using an IAM role for authentication.
[12:34 PM PST] We continue to experience increased API error rates for multiple AWS Services in the US-EAST-1 Region. The root cause of this issue is an impairment of several network devices. We continue to work toward mitigation, and are actively working on a number of different mitigation and resolution actions. While we have observed some early signs of recovery, we do not have an ETA for full recovery. For customers experiencing issues signing-in to the AWS Management Console in US-EAST-1, we recommend retrying using a separate Management Console endpoint (such as https://us-west-2.console.aws.amazon.com/). Additionally, if you are attempting to login using root login credentials you may be unable to do so, even via console endpoints not in US-EAST-1. If you are impacted by this, we recommend using IAM Users or Roles for authentication. We will continue to provide updates here as we have more information to share.
[2:04 PM PST] We have executed a mitigation which is showing significant recovery in the US-EAST-1 Region. We are continuing to closely monitor the health of the network devices and we expect to continue to make progress towards full recovery. We still do not have an ETA for full recovery at this time.
[2:47 PM PST] We have mitigated the underlying issue that caused some network devices in the US-EAST-1 Region to be impaired. We are seeing improvement in availability across most AWS services. All services are now independently working through service-by-service recovery. We continue to work toward full recovery for all impacted AWS Services and API operations. In order to expedite overall recovery, we have temporarily disabled Event Deliveries for Amazon EventBridge in the US-EAST-1 Region. These events will still be received & accepted, and queued for later delivery.
Are your apps, smart home devices, and other Amazon connected services working again?
You may be required to sign in again and in some cases reset links between home automation products such as lights, doorbells, security cameras and air condition controllers.
If you use Alexa to turn off and on items at home, follow these steps to make sure things are back to normal once AWS service outages are restored.
How to Check Smart Home Devices on Amazon Alexa
- Launch Alexa app on your mobile device
- Tap the devices icon at the bottom of the screen
- Make sure smart devices you are checking are plugged in and turned on even if you must power them on manually or by using the companion app from the manufacturer
- On the Alexa app devices screen, swipe toward the left where it starts with ‘Echo & Alexa’ until you find ‘All Devices’
- Read through the list of devices to see the status of each device. Stop at each device that says ‘Device is unresponsive’
- Try to power it on or interact with it via Alexa app and if still unresponsive, check that the makers name by clicking the Settings icon in the upper right part of the screen.
Usually one of two things is creating the trouble
1) The smart device skill may need to be unlinked and linked again so that permission for Alexa to connect to the smart device makers login credentials are restored
2) you may to reinstall the smart device again
Here’s how to troubleshoot the Skill in the Alexa app
- Within the Alexa mobile app, tap the three line menu on the bottom right of the screen
- Tap Skills & Games from the menu
- Tap Your Skills and scroll until you see the smart device maker’s app name.
- Make sure you have the login and password for each device maker’s app
- Tap the skill and then choose to DISABLE SKILL to unlink your account from Alexa
- Reinstall the smart device skill to link your accounts to Alexa.