Incomplete SSL Certificate Propagation
Incident Report for Ello
We recently replaced our soon-to-expire www.ello.co SSL certificate with a new wildcard certificate (*.ello.co). Because we use a bot detection service, the certificate needed to be installed on their points of presence (POP's) around the globe. Unfortunately, they missed one. And we missed that they missed it. When our older certificate expired (about 29:30 GMT), users being routed through the POP without the new certificate got an SSL error instead of the wonderful Ello goodness they wanted. Fortunately, Ello has an amazing user community. A couple of key users affected by the issue helped us troubleshoot (HUGE thanks to them!), and we were able to resolve the issue in 3-4 hours. We are planning to implement more comprehensive monitoring that will catch this type of issue proactively in the future.
Posted almost 2 years ago. Mar 29, 2015 - 15:30 EDT