Monitoring an issue with CA for Cloud PKI for Trusted Endpoints
Postmortem
Issuing or Renewing Certificates - All Deployments
Incident Report - 2019/08/30
Summary
From 2:30 p.m. EDT to 2:56 p.m. EDT on August 30, 2019, Duo customers were unable to issue or renew Trusted Endpoints certificates.
Details
Duo’s Engineering Team was alerted by our monitoring platform at 2:35 p.m. of an increased error rate in our issuing and renewing of certificates from our cloud PKI service. Engineering observed an issue with the firewall protecting the cloud PKI service, restarted the firewall, and continued to monitor the error rate until the problem subsided at 2:56 p.m.
After analysis, it was determined that the firewall in our cloud PKI experienced an out-of-memory condition which caused it to stop routing traffic. We will be following up with the firewall vendor and investigating additional solutions to prevent this failure going forward.
Posted Sep 04, 2019 - 16:45 EDT
Resolved
We are resolving this issue, as we believe the vendor has completely remediated the issue and our monitoring does not indicate that the issue will resurface. A formal RCA will be posted shortly.
Posted Aug 30, 2019 - 16:04 EDT
Update
We are continuing to monitor for any further issues.
Posted Aug 30, 2019 - 15:05 EDT
Update
We are continuing to monitor for any further issues.
Posted Aug 30, 2019 - 15:05 EDT
Monitoring
One of our certificate providers have notified us of an issue that may have affected some customers when issuing or renewing certificates for our Cloud PKI for Trusted Endpoints. The issue has been remediated but we are monitoring for any customers who may have been affected. Authentications were not affected in any way.
Posted Aug 30, 2019 - 15:03 EDT
This incident affected: DUO1 (Cloud PKI), DUO2 (Cloud PKI), DUO3 (Cloud PKI), DUO4 (Cloud PKI), DUO5 (Cloud PKI), DUO6 (Cloud PKI), DUO7 (Cloud PKI), DUO8 (Cloud PKI), DUO10 (Cloud PKI), DUO9 (Cloud PKI), DUO11 (Cloud PKI), DUO12 (Cloud PKI), DUO13 (Cloud PKI), DUO15 (Cloud PKI), DUO14 (Cloud PKI), DUO16 (Cloud PKI), DUO17 (Cloud PKI), DUO18 (Cloud PKI), DUO19 (Cloud PKI), DUO20 (Cloud PKI), DUO21 (Cloud PKI), DUO22 (Cloud PKI), DUO23 (Cloud PKI), DUO24 (Cloud PKI), DUO25 (Cloud PKI), DUO26 (Cloud PKI), DUO27 (Cloud PKI), DUO28 (Cloud PKI), DUO29 (Cloud PKI), DUO30 (Cloud PKI), DUO31 (Cloud PKI), DUO32 (Cloud PKI), DUO33 (Cloud PKI), DUO34 (Cloud PKI), DUO36 (Cloud PKI), DUO37 (Cloud PKI), DUO38 (Cloud PKI), DUO39 (Cloud PKI), DUO40 (Cloud PKI), DUO41 (Cloud PKI), DUO42 (Cloud PKI), DUO43 (Cloud PKI), DUO44 (Cloud PKI), DUO45 (Cloud PKI), DUO46 (Cloud PKI), DUO47 (Cloud PKI), DUO48 (Cloud PKI), DUO49 (Cloud PKI), DUO50 (Cloud PKI), DUO51 (Cloud PKI), DUO52 (Cloud PKI), DUO53 (Cloud PKI), DUO54 (Cloud PKI), DUO55 (Cloud PKI), DUO56 (Cloud PKI), DUO57 (Cloud PKI), DUO58 (Cloud PKI), DUO59 (Cloud PKI), DUO60 (Cloud PKI), DUO61 (Cloud PKI), DUO62 (Cloud PKI), DUO63 (Cloud PKI), DUO64 (Cloud PKI), DUO65 (Cloud PKI), and DUO35 (Cloud PKI).