On Jan 21 from roughly 6:25 - 7:36pm EST, Docs SSL certificate validation began to fail for . The incident was caused by an issue with our upstream hosting provider, which then caused an OCSP check that we use to talk with the CA (Certificate Authority) to fail. By temporarily disabling the OCSP check, we were able to restore service back to normal.
The issue drew out longer than we’d like because we tried a number of things before finally tracking down the root cause. We always look at our own infrastructure first, and since this incident was caused by an external event that was not mentioned publicly, it took more time to resolve the problem.
Our upstream provider seems confident the problem is permanently resolved, as we’ve been told they “would be very surprised if this ever reoccurs.” There isn’t much we could have done on our end to prevent this one, but we have updated our internal procedures so that we file a ticket with the upstream provider immediately, instead of waiting to diagnose a problem for ourselves.
Sorry again for the trouble on Monday, but the good news is that we don’t expect to ever see the same issue again. If so, we’ll better know how to resolve it.