All Deployments: Intermittent SMS Delivery Issues
Incident Report for Duo
Postmortem

Summary

On Monday, August 31, 2020 from approximately 1:15 PM EDT to 3:40 PM EDT, customers may have experienced intermittent issues receiving SMS messages containing authentication passcodes. All other authentication methods were unaffected. Most customers were able to request and successfully receive SMS passcodes after retrying. Approximately 5,400 SMS messages were affected. 

Details

At 1:00 PM EDT, Duo’s automated monitoring detected an intermittent connectivity issue with our primary SMS provider. Duo’s Engineering team immediately began investigating the issue. Duo uses multiple telephony providers to support different regions across the globe. Duo’s platform will automatically attempt to deliver SMS messages via a secondary provider if the primary provider fails.

At 1:15 PM EDT, the Engineering team failed over US SMS traffic to a secondary provider in hopes of reducing delays in SMS delivery introduced by the connectivity issues with our primary provider. The secondary provider was unable to handle the entire volume of US SMS traffic and Duo’s Engineering team quickly moved traffic back to the primary provider. Unfortunately, the automation used to move between providers contained a bug. This bug meant that when traffic was moved back to the primary provider, the secondary provider was not correctly configured to automatically provide backup service. 

At 3:40 PM, Duo’s Engineering team completed configuration changes to once again allow the platform to attempt SMS delivery via our secondary provider when the primary fails. Using our primary SMS provider with a functional backup was sufficient to fully resolve the delivery issues.

Duo’s Engineering team has fixed our automation so that the configuration issue cannot happen again. Additionally we are working with our secondary SMS provider to ensure they can handle 100% of our global SMS traffic. Duo is also committed to onboarding additional telephony providers to provide additional redundancy.

Posted Sep 01, 2020 - 09:48 EDT

Resolved
Following a monitoring period, we have confirmed that the issues impacting SMS delivery are now fully resolved.

We will be posting a Root-Cause Analysis (RCA) to this Incident as soon as it is available.
Posted Aug 31, 2020 - 17:14 EDT
Monitoring
The issues impacting SMS delivery are now resolved, and we are continuing to monitor for any additional occurrences of the behavior now that a fix is in place.

We will be posting a Root-Cause Analysis (RCA) to this Incident as soon as it is available.
Posted Aug 31, 2020 - 16:12 EDT
Identified
We have identified the cause of issue with intermittent delivery for some SMS messages on all of our deployments and are working to correct the issue as soon as possible.

We suggest using other authentication methods such as Duo Push, Phone Callback, or Passcodes, or trying to send an SMS passcode again, as a workaround.

Please check back here or subscribe to updates for any changes.
Posted Aug 31, 2020 - 14:00 EDT
Investigating
We are currently investigating an issue causing intermittent delivery issues with some SMS messages on all of our deployments and are working to correct the issue as soon as possible.

We suggest using other authentication methods such as Duo Push, Phone Callback, or Passcodes, or trying to send an SMS passcode again, as a workaround.

Please check back here or subscribe to updates for any changes.
Posted Aug 31, 2020 - 13:48 EDT
This incident affected: DUO7 (SMS Message Delivery), DUO46 (SMS Message Delivery), DUO33 (SMS Message Delivery), DUO1 (SMS Message Delivery), DUO21 (SMS Message Delivery), DUO44 (SMS Message Delivery), DUO51 (SMS Message Delivery), DUO14 (SMS Message Delivery), DUO30 (SMS Message Delivery), DUO36 (SMS Message Delivery), DUO26 (SMS Message Delivery), DUO38 (SMS Message Delivery), DUO61 (SMS Message Delivery), DUO29 (SMS Message Delivery), DUO60 (SMS Message Delivery), DUO19 (SMS Message Delivery), DUO48 (SMS Message Delivery), DUO55 (SMS Message Delivery), DUO9 (SMS Message Delivery), DUO63 (SMS Message Delivery), DUO27 (SMS Message Delivery), DUO3 (SMS Message Delivery), DUO50 (SMS Message Delivery), DUO8 (SMS Message Delivery), DUO40 (SMS Message Delivery), DUO58 (SMS Message Delivery), DUO43 (SMS Message Delivery), DUO57 (SMS Message Delivery), DUO32 (SMS Message Delivery), DUO24 (SMS Message Delivery), DUO2 (SMS Message Delivery), DUO28 (SMS Message Delivery), DUO10 (SMS Message Delivery), DUO12 (SMS Message Delivery), DUO5 (SMS Message Delivery), DUO16 (SMS Message Delivery), DUO41 (SMS Message Delivery), DUO6 (SMS Message Delivery), DUO13 (SMS Message Delivery), DUO18 (SMS Message Delivery), DUO37 (SMS Message Delivery), DUO23 (SMS Message Delivery), DUO20 (SMS Message Delivery), DUO15 (SMS Message Delivery), DUO49 (SMS Message Delivery), DUO56 (SMS Message Delivery), DUO59 (SMS Message Delivery), DUO52 (SMS Message Delivery), DUO39 (SMS Message Delivery), DUO47 (SMS Message Delivery), DUO22 (SMS Message Delivery), DUO17 (SMS Message Delivery), DUO53 (SMS Message Delivery), DUO45 (SMS Message Delivery), DUO34 (SMS Message Delivery), DUO42 (SMS Message Delivery), DUO31 (SMS Message Delivery), DUO62 (SMS Message Delivery), DUO4 (SMS Message Delivery), DUO54 (SMS Message Delivery), DUO65 (SMS Message Delivery), DUO11 (SMS Message Delivery), DUO64 (SMS Message Delivery), DUO25 (SMS Message Delivery), and DUO35 (SMS Message Delivery).