Starting around 3:55PM EST on December 20th, we started to investigate reports of folder views, Workflows, tags, search results, and reporting drill downs failing to update in a timely manner.
After investigation, we found that the root cause of the delayed updates was due to our search indexer losing connection to our message queueing service. Both components are needed to continuously process conversation updates. The connection between the indexer and the message queuing service ultimately failed due to faulty queue policy that was applied manually, rather than applied through automated channels, which is our standard process.
In most cases, we’d expect to see various alerts when something like this happens. Regretfully, this was a strange edge case that didn't generate any of the alerts that we’d typically encounter when there’s an issue with the search indexer.
In order to resolve the root issue, we needed to stop, start, and rebuild services to stabilize the indexer. Once the queue policy was addressed and connections restored, all services returned to normal.
We identified a specific time period between 2:40PM EST and 5:00PM EST where conversation updates and events were not processed as they should have been, due to the lack of connection between the indexer and the message queuing service. For example, during this time period:
No data was lost during this event. With the exception of Workflow-based emails, which were a bit delayed, all inbound and outbound messages were sent and delivered.
After services were stabilized, we started to re-queue all messages and events that took place between 2:40PM EST and 5:00PM EST. In most cases, we’d expect this sort of recovery to take a few hours.
However, due to the amount of data we needed to work through, and due to additional load placed on the indexer from a separate service we had running at the time, the re-queue process took longer than expected. As of 4:00AM EST on Friday December 22nd, the re-queue process was complete.
Although we don’t expect to see a similar issue arise again, we’re working to identify any gaps in our monitoring and alerting services so that we’ll be able to catch a similar event a bit faster. We’re also going to change the way we handle policy updates behind the scenes, so that they do not affect the search and indexing services that are currently running.
We do everything we can do avoid these moments, and we’re sorry that this caused so much trouble and confusion.