[DUO34] Full Deployment Outage

Incident Report for Duo

Postmortem

DUO34 Deployment Outage 

Incident Report – 10/7/2025 

Summary 

On October 7, 2025, between 2:40pm EDT and 3:40pm EDT, all services on Duo 34 were unavailable. This was due to a misconfiguration in our edge load balancer which stopped all traffic to that deployment. Duo engineering promptly fixed the misconfiguration and have fixed the underlying cause, to ensure our systems are more resilient. 

Details 

Duo’s application platform has two parts to manage our services. One-part handles creating the network setup, and the other controls how traffic reaches running services. A service disruption occurred because of a misalignment between these two key systems. Even though the network was set up correctly, changes were not reflected in the traffic routing settings. This caused traffic to stop reaching the services, resulting in the outage.   

To restore service, a manual fix was put in place to connect these network servers with the right settings. As a long-term solution, we have updated the system settings to point to the right destinations. This will ensure that the misalignment will not happen again. To avoid this problem in the future, we're reviewing how we manage system configurations, improving our setup process to avoid local files, and adding checks to stop changes that could cause similar issues.

Posted Oct 09, 2025 - 10:05 EDT

Resolved

The issue causing an outage for all services on our DUO34 deployment has been resolved.

We will publish a RCA as soon as it is available. You can check back here for the latest information or subscribe for updates.
Posted Oct 07, 2025 - 15:56 EDT

Monitoring

We have deployed a fix for the issue causing an outage for all services on our DUO34 deployment and are monitoring the results at this time.
Posted Oct 07, 2025 - 15:44 EDT

Identified

We have identified the issue causing an outage for all services on our DUO34 deployment and working to deploy a fix.
Posted Oct 07, 2025 - 15:23 EDT

Investigating

We are currently investigating an issue causing a full outage for all services on our DUO34 deployment and are working to restore services as soon as possible.

Please check back here or subscribe to updates for any changes
Posted Oct 07, 2025 - 15:13 EDT
This incident affected: DUO34 (Core Authentication Service, Admin Panel, Push Delivery, Phone Call Delivery, SMS Message Delivery).