All Deployments: Intermittent SMS Delivery Delays to US Phone Numbers
Postmortem
Incident Report - 2020-12-1
Summary
On Monday, November 30, 2020 from 12:40 p.m. EST to 7:18 p.m. EST, a small number of customers with US phone numbers experienced intermittent issues receiving SMS messages containing Duo authentication passcodes. No other authentication methods were affected. Duo switched to a secondary telephony provider to deliver SMS messages. Our primary provider resolved issues and Duo returned to normal service by 7:30 p.m. EST.
Details
Beginning at 12:40 p.m. EST on Monday, November 30, 2020, Duo’s monitoring recorded a small number of failures to deliver passcodes via SMS. At 1:54 p.m. EST, Duo customers reported delays in receiving SMS passcodes.
Based on monitoring and customer reports of delayed messages, we made the decision at 2:02 p.m. EST to deliver SMS messages solely via our secondary provider. At 7:18 p.m. EST, our primary service provider reported that the issues with SMS delivery delays had been investigated with their carrier partners and fixed. Duo’s Engineering Team confirmed that service had been restored.
At 7:30 p.m. EST, we switched back to delivering SMS passcodes via our primary service provider.
Duo’s Engineering Team is working with our primary service provider to gain better visibility into SMS delivery delays in the future in order to identify issues faster.
Posted Dec 01, 2020 - 13:40 EST
Resolved
The issues with intermittent SMS Message Delivery delays to US Phone Numbers on all of our deployments are fully resolved and all services are now fully functional.
We will be posting a root-cause analysis (RCA) in just a moment.
Posted Dec 01, 2020 - 13:22 EST
Monitoring
The issues impacting SMS delivery delays to US phone numbers are now resolved, and we are continuing to monitor for any additional occurrences of the behavior.
We will be posting a Root-Cause Analysis (RCA) to this Incident as soon as it is available.
Posted Nov 30, 2020 - 20:58 EST
Update
We have confirmed this issue is related to intermittent SMS delivery delays from one of our telephony providers. We have failed over to an additional provider until service is restored and are monitoring delivery of SMS passcodes.
If you are still experiencing an issue receiving SMS passcodes, we suggest using other authentication methods such as Duo Push, phone callback, or passcodes, or try sending an SMS passcode again, as a workaround.
Please check back here for more information or subscribe to updates.
Posted Nov 30, 2020 - 15:46 EST
Identified
We have identified an issue causing intermittent delivery delay issues with some SMS messages to US Phone Numbers on all of our deployments and are working to correct the issue as soon as possible.
We suggest using other authentication methods such as Duo Push, Phone Callback, or Passcodes, or trying to send an SMS passcode again, as a workaround.
Please check back here or subscribe to updates for any changes.
Posted Nov 30, 2020 - 14:10 EST
This incident affected: DUO1 (SMS Message Delivery), DUO2 (SMS Message Delivery), DUO3 (SMS Message Delivery), DUO4 (SMS Message Delivery), DUO5 (SMS Message Delivery), DUO6 (SMS Message Delivery), DUO7 (SMS Message Delivery), DUO8 (SMS Message Delivery), DUO47 (SMS Message Delivery), DUO10 (SMS Message Delivery), DUO11 (SMS Message Delivery), DUO12 (SMS Message Delivery), DUO13 (SMS Message Delivery), DUO14 (SMS Message Delivery), DUO15 (SMS Message Delivery), DUO16 (SMS Message Delivery), DUO17 (SMS Message Delivery), DUO18 (SMS Message Delivery), DUO19 (SMS Message Delivery), DUO20 (SMS Message Delivery), DUO21 (SMS Message Delivery), DUO22 (SMS Message Delivery), DUO23 (SMS Message Delivery), DUO24 (SMS Message Delivery), DUO25 (SMS Message Delivery), DUO26 (SMS Message Delivery), DUO27 (SMS Message Delivery), DUO28 (SMS Message Delivery), DUO29 (SMS Message Delivery), DUO30 (SMS Message Delivery), DUO31 (SMS Message Delivery), DUO32 (SMS Message Delivery), DUO33 (SMS Message Delivery), DUO34 (SMS Message Delivery), DUO36 (SMS Message Delivery), DUO37 (SMS Message Delivery), DUO38 (SMS Message Delivery), DUO39 (SMS Message Delivery), DUO40 (SMS Message Delivery), DUO41 (SMS Message Delivery), DUO42 (SMS Message Delivery), DUO43 (SMS Message Delivery), DUO44 (SMS Message Delivery), DUO45 (SMS Message Delivery), DUO46 (SMS Message Delivery), DUO48 (SMS Message Delivery), DUO9 (SMS Message Delivery), DUO49 (SMS Message Delivery), DUO50 (SMS Message Delivery), DUO51 (SMS Message Delivery), DUO52 (SMS Message Delivery), DUO53 (SMS Message Delivery), DUO54 (SMS Message Delivery), DUO55 (SMS Message Delivery), DUO56 (SMS Message Delivery), DUO57 (SMS Message Delivery), DUO58 (SMS Message Delivery), DUO59 (SMS Message Delivery), DUO60 (SMS Message Delivery), DUO61 (SMS Message Delivery), DUO62 (SMS Message Delivery), DUO63 (SMS Message Delivery), DUO64 (SMS Message Delivery), DUO65 (SMS Message Delivery), DUO66 (SMS Message Delivery), and DUO35 (SMS Message Delivery).