Search and Folder issues

Incident Report for Help Scout

Postmortem

On Feb 7, we experienced a service issue in our search and reports infrastructure, resulting in some delays in updates to search results, folder counts, message previews, email and workflow processing, and reports. During this time, Help Scout experienced no downtime in the web or mobile apps nor was any data lost.

Folder count updates, message previews and workflows were delayed for roughly an hour, while report queues took about 3 hours to normalize.

What can we do better?

Moving forward, we’ll be more vigilant in actively monitoring the segment of our search infrastructure backend which caused the delays this morning. With the current fixes in place, we’re better positioned to ensure they don’t occur again ahead of future upgrades to search infrastructure. Sorry again for the trouble this morning folks.

Posted Feb 19, 2019 - 15:18 EST

Resolved

We're all clear! All queue backups (Reports + Search) have now been resolved. Very sorry for any interruption today; we'll publish a postmortem with more detail on what happened in the next 48 hours.

Posted Feb 07, 2019 - 05:26 EST

Monitoring

We've deployed some changes, and Reports queues are dropping back to normal levels. There was a brief spike in our Search queue due to this change, but we're seeing improvement there in line with our Reports queue.

We'll keep this status open to monitor a little while longer, but everything should be back to normal shortly. We'll be in touch with what's hopefully our final update soon.

Posted Feb 07, 2019 - 05:17 EST

Update

Our team have identified a fix for our outstanding Reports issue this morning; all other functions are operating normally. We'll have an update available within the hour on the progress of the Reports-fix rollout, but we'll keep you in the loop if anything changes with that timeline. Thanks for your patience this morning.

Posted Feb 07, 2019 - 04:21 EST

Identified

We have identified and are addressing the issue. Conversation search results and folders counts should be catching up. Currently, only reports are still delayed. More updates to come.

Posted Feb 07, 2019 - 03:42 EST

Investigating

We're seeing some delays with folder counts, message previews and workflows. Our team is investigating and we're hoping to have this resolved shortly. More updates as we have them.

Posted Feb 07, 2019 - 03:34 EST

This incident affected: Mailbox counts, folders, reports, search results.