On December 4, 2023, at around 14:57 EST, Duo's Engineering Team was alerted by Support that a handful of administrators were unable to login to the admin panel. The root cause was identified as connectivity issues from certain internet service providers (ISPs).
The issue was resolved on December 5, 2023 8:14 EST.
2023-12-04 14:57 Duo Site Reliability Engineering (SRE) is informed by Duo Customer Support (CS) that four customers are reporting issues with admin panel login. SRE begins triage.
2023-12-04 15:22 SRE escalated to the admin platform engineering team to investigate.
2023-12-04 15:29 We concluded that the impact is limited to only a few admin panel users.
2023-12-04 15:37 Customer Support received an additional impacted user.
2023-12-04 16:17 Status page updated to Investigating.
2023-12-04 16:37 Engineers requested more information (username, account and ISP) from impacted customers via CS.
2023-12-04 16:59 Duo successfully tested the issue from the Aus/NZ network using two different networks and VPN endpoints.
2023-12-04 17:17 Status page updated to Monitoring.
2023-12-05 07:44 Engineers discovered no issue from impacted customers’ traffic on our monitoring system.
2023-12-05 08:26 Status page updated to Resolved.
Duo SRE was notified about an issue of a user not able to login to the admin panel. We received four such users in a span of a few hours. Duo engineers diligently analyzed the monitoring systems, and attempted to reproduce the issue from various internet routes but failed to reproduce the issue. During investigation, we found the issue was occurring with certain ISPs.
How did Duo Resolve the incident
Duo analyzed the monitoring systems, ruled out various possibilities by attempting to reproduce the issue. All efforts led us to the network connectivity issue for the impacted user.
How many Customers were impacted, if applicable
Few administrators from four customers.
What is Duo doing to prevent this in the future?
The network connectivity issues at the user's premise is not under Duo’s control. The network related issues can be tested and validated by using different internet routes which are already incorporated in the triaging process.
Note: You can find your Duo deployment’s ID and sign up for updates via the StatusPage by following the instructions in this knowledge base article.