S3 connection issues

Incident Report for Help Scout

Resolved

We have about 110k emails left to process and expect to be all caught up in about an hour. Otherwise we're in the clear and are closing out this issue. Thanks so much for the patience today!
Posted Feb 28, 2017 - 17:56 EST

Monitoring

We haven't seen an update from Amazon yet, but incoming mail and attachment uploads seem to be working nicely now! We're processing what we can and will let you know when the queues are all clear.
Posted Feb 28, 2017 - 17:08 EST

Update

Viewing attachments, Docs assets, and all functionality relevant to Beacon should be set now. We're still waiting on updates that allow us to process incoming emails and upload attachments, basically anything that adds new files to S3. That's progress!
Posted Feb 28, 2017 - 16:25 EST

Update

We're starting to see the light at the end of the tunnel. Based on Amazon's latest update, we're hoping to see improvements within the hour.
Posted Feb 28, 2017 - 15:57 EST

Update

We're still holding tight while Amazon continues to work on restoring services. No ETA just yet.
Posted Feb 28, 2017 - 15:03 EST

Update

Amazon has identified the problem and is actively working on a fix. No ETA was given, but we'll keep watching.
Posted Feb 28, 2017 - 14:02 EST

Update

S3 errors persist and we're waiting on more information. It's likely that most web services you use are impacted by this event. Even our status page doesn't look right! We hope to be back on track soon.
Posted Feb 28, 2017 - 13:21 EST

Identified

We're currently monitoring an issue with the service we use for file storage: Amazon S3. The issue is impacting inbound email processing, attachments and Docs assets (CSS, images). We're monitoring it closely and will post updates as we have them from Amazon.
Posted Feb 28, 2017 - 13:03 EST