Starting around 5:54pm EDT on Monday evening, a failure with one of our search nodes caused degraded performance with the web app, the public help desk API, as well as Docs search.
Performance issues lasted for 43 minutes, up to 6:37pm EDT. During this time, the web app was down and unavailable for 9 minutes. Customers experienced slow page loads/time outs when attempting to access Help Scout, as well as failed requests to the API.
When the node failed, our search API also failed. The web app as well as Docs rely heavily on the search API, which caused the underlying performance issues with those services.
This root of the issue was tough to track down initially. To resolve the problem, we restarted the failed search node, which helped all services to recover.
We have multiple search nodes in place, but one node going down should not negatively impact other services. This isn't an issue we've encountered in the past, so we still have a more investigative work to do on our end to figure out what caused the node, along with the search API, to fail in the first place. Based on what we find, we'll take all steps necessary to prevent this issue from happening again.