DUO1: Authentication and Admin Panel Access Issues
Incident Report for Duo
Postmortem

Authentication and Admin Panel Access Outage - DUO1

Summary

On February 25, 2021 from 2:35 a.m. EST to 3:21 a.m. EST, Duo's Cloud Service experienced issues affecting the DUO1 deployment which resulted in end-user authentication failures for some customers. Some customers were also unable to access the Duo Admin Panel via admin.duosecurity.com. The Duo Engineering Team was able to fully resolve this network issue at 3:21 a.m. EST.

Details

At 2:35 a.m. EST on February 25, the Duo Engineering Team was conducting routine maintenance on our DUO1 deployment. A previously undiscovered configuration defect prevented our load balancer failover logic from working normally, resulting in a service load balancer becoming unreachable from the public internet. This resulted in some customers being unable to complete Duo-protected authentication or access the Duo Admin Panel.

Proactive service monitoring immediately alerted the Duo Engineering Team to these issues, and we were able to begin working on a solution at 2:36 a.m. EST. At 3:21 a.m. EST, we were able to restore our load balancing solution, and both authentication and administration issues were fully resolved.

The Duo Engineering Team is currently identifying other potential configuration defects so this failover logic error does not cause a service issue again. We are committed to identifying additional process improvements so we can deliver an even more reliable, resilient service moving forward.

Posted Feb 25, 2021 - 13:25 EST

Resolved
Our Engineering team has completed monitoring of the incident and confirmed that services and authentications have returned to normal. We have not seen any evidence of ongoing issues. We will continue working to improve the reliability of our service, and an RCA of this issue will be published here shortly.
Posted Feb 25, 2021 - 04:59 EST
Monitoring
Our Engineering team has identified the issue and a fix has been implemented. The issue should now be resolved, and we're monitoring the situation.
Posted Feb 25, 2021 - 03:43 EST
Identified
Customers have reported degraded services and authentication on some DUO1 tenants. Our Engineering team is currently working to resolve the issue. Please subscribe here for further updates.
Posted Feb 25, 2021 - 03:30 EST
This incident affected: DUO1 (Core Authentication Service, Admin Panel, Push Delivery, Phone Call Delivery, SMS Message Delivery, Cloud PKI, SSO).