At Help Scout we use search very heavily to index large amounts of data. When you view the Closed folder, for instance, behind-the-scenes the results are returned from a search index and not directly from the database. We use search in a similar way for dozens of things.

This morning we were testing some improvements that would persist changes to our search indexes faster than it currently works. We tested the changes in our development environment and it went great.

When we put the changes into production, the search cluster locked up because there was too much data hitting it at once. We instantly realized the mistake and it took roughly 16 minutes to get everything back to normal.

Hindsight being 20/20, we understand exactly how this should have been implemented and won't make the same mistake again. Very sorry for the trouble today!