Today from 11:32 - 11:52 EST, Help Scout users experienced somewhat elevated page load times – and in some cases page load failures – in the web application. Mailbox APIs 1.0 and 2.0 also had elevated response times and some failures.
The root cause of the issue was unrelated to the few performance issues we faced last week. In this case, a drastic increase in email traffic (did someone say Cyber Monday?) created a chain reaction that caused Help Scout to perform slowly for the 20-minute window. The Ops team was able to increase available resources and handle the load eventually, resolving the problem.
We haven’t seen an issue like this one before. Fortunately there are a couple of software upgrades we can make, and one configuration change that will be made to our message queueing service. Both should make our systems more resilient if this situation were to happen again.
We understand how critical it is for Help Scout to be on point for your team during the holiday season, and we’re up to the task. Very sorry for the trouble today, but we’re poised to make sure it does not recur.