Search and Folder issues
Incident Report for Help Scout
Postmortem

On Feb 7, we experienced a service issue in our search and reports infrastructure, resulting in some delays in updates to search results, folder counts, message previews, email and workflow processing, and reports. During this time, Help Scout experienced no downtime in the web or mobile apps nor was any data lost.

Folder count updates, message previews and workflows were delayed for roughly an hour, while report queues took about 3 hours to normalize.

What can we do better?

Moving forward, we’ll be more vigilant in actively monitoring the segment of our search infrastructure backend which caused the delays this morning. With the current fixes in place, we’re better positioned to ensure they don’t occur again ahead of future upgrades to search infrastructure. Sorry again for the trouble this morning folks.

Posted 6 months ago. Feb 19, 2019 - 15:18 EST

Resolved
We're all clear! All queue backups (Reports + Search) have now been resolved. Very sorry for any interruption today; we'll publish a postmortem with more detail on what happened in the next 48 hours.
Posted 6 months ago. Feb 07, 2019 - 05:26 EST
Monitoring
We've deployed some changes, and Reports queues are dropping back to normal levels. There was a brief spike in our Search queue due to this change, but we're seeing improvement there in line with our Reports queue.

We'll keep this status open to monitor a little while longer, but everything should be back to normal shortly. We'll be in touch with what's hopefully our final update soon.
Posted 6 months ago. Feb 07, 2019 - 05:17 EST
Update
Our team have identified a fix for our outstanding Reports issue this morning; all other functions are operating normally. We'll have an update available within the hour on the progress of the Reports-fix rollout, but we'll keep you in the loop if anything changes with that timeline. Thanks for your patience this morning.
Posted 6 months ago. Feb 07, 2019 - 04:21 EST
Identified
We have identified and are addressing the issue. Conversation search results and folders counts should be catching up. Currently, only reports are still delayed. More updates to come.
Posted 6 months ago. Feb 07, 2019 - 03:42 EST
Investigating
We're seeing some delays with folder counts, message previews and workflows. Our team is investigating and we're hoping to have this resolved shortly. More updates as we have them.
Posted 6 months ago. Feb 07, 2019 - 03:34 EST
This incident affected: Mailbox counts, folders, reports, search results.