Docs SSL errors
Incident Report for Help Scout
Postmortem

What happened

On Jan 21 from roughly 6:25 - 7:36pm EST, Docs SSL certificate validation began to fail for . The incident was caused by an issue with our upstream hosting provider, which then caused an OCSP check that we use to talk with the CA (Certificate Authority) to fail. By temporarily disabling the OCSP check, we were able to restore service back to normal.

The issue drew out longer than we’d like because we tried a number of things before finally tracking down the root cause. We always look at our own infrastructure first, and since this incident was caused by an external event that was not mentioned publicly, it took more time to resolve the problem.

What can we do better?

Our upstream provider seems confident the problem is permanently resolved, as we’ve been told they “would be very surprised if this ever reoccurs.” There isn’t much we could have done on our end to prevent this one, but we have updated our internal procedures so that we file a ticket with the upstream provider immediately, instead of waiting to diagnose a problem for ourselves.

Sorry again for the trouble on Monday, but the good news is that we don’t expect to ever see the same issue again. If so, we’ll better know how to resolve it.

Posted 9 months ago. Jan 23, 2019 - 10:40 EST

Resolved
All SSL errors have been resolved. Very sorry for the interruption today! We'll publish a postmortem with much more detail in the next 48 hours.
Posted 9 months ago. Jan 21, 2019 - 19:41 EST
Identified
We're in the process of patching the SSL issues up now, and docs.helpscout.com is securely serving up content again. Should be all set soon.
Posted 9 months ago. Jan 21, 2019 - 19:36 EST
Update
Really sorry for the trouble, folks. We're still working on this.
Posted 9 months ago. Jan 21, 2019 - 19:18 EST
Update
Still investigating, we're close to understanding the root cause and getting everything back to normal.
Posted 9 months ago. Jan 21, 2019 - 18:59 EST
Investigating
We're investigating SSL errors on Docs sites with custom domains. This includes our own site at docs.helpscout.com. More as we figure out what's up ...
Posted 9 months ago. Jan 21, 2019 - 18:25 EST
This incident affected: Docs Sites.