All Deployments: SMS Delivery Degradation to some US numbers
Postmortem
Degraded SMS Delivery to the US - All Deployments
Incident Report - 2019/12/05
Summary
From 11:44 AM to 1:45 PM EDT on Thursday, December 5, SMS messages from Duo to US numbers intermittently failed to deliver. This was due to issues that one of Duo’s downstream telephony providers experienced via a carrier partner. The provider resolved the issue before Duo finalized routing all affected customers through a backup provider.
Details
Duo’s platform routes SMS messages among multiple partners and carriers to enable SMS passcode delivery and other SMS workflows. One of Duo's primary providers experienced issues delivering SMS messages with one of their carrier partners on December 5.
Upon notification of the issue, Duo's engineering team began preparations to move affected customers to a secondary provider. As testing was under way to reroute all affected customers, the issue was resolved.
The engineering team is working to add additional telephony providers to the service as well as developing faster automated failover solutions.
We will update this post with additional details when they are provided by our telephony partner.
Posted Dec 10, 2019 - 12:21 EST
Resolved
The issue regarding SMS Message Delivery on our deployments is fully resolved and all services are now fully functional.
We will be posting a root-cause analysis (RCA) here once our engineering team has finished its thorough investigation of the issue.
Please make sure to check back or subscribe to be notified when the RCA is posted.
Posted Dec 05, 2019 - 14:07 EST
Monitoring
Our telephony provider is seeing recovery on the SMS delivery degradation, and we have confirmation that customers are receiving SMS passcodes successfully.
We will continue to monitor the issue and will post any updates when the incident is considered fully resolved.
Please check back here or subscribe here for further updates.
Posted Dec 05, 2019 - 13:31 EST
Update
We are continuing to investigate this issue.
Posted Dec 05, 2019 - 12:37 EST
Investigating
We are currently investigating an issue causing failures with SMS Message Delivery to some US numbers on all of our deployments.
Please check back here or subscribe to updates for any changes.
Posted Dec 05, 2019 - 12:32 EST
This incident affected: DUO1 (SMS Message Delivery), DUO2 (SMS Message Delivery), DUO3 (SMS Message Delivery), DUO4 (SMS Message Delivery), DUO5 (SMS Message Delivery), DUO6 (SMS Message Delivery), DUO7 (SMS Message Delivery), DUO8 (SMS Message Delivery), DUO47 (SMS Message Delivery), DUO10 (SMS Message Delivery), DUO11 (SMS Message Delivery), DUO12 (SMS Message Delivery), DUO13 (SMS Message Delivery), DUO14 (SMS Message Delivery), DUO15 (SMS Message Delivery), DUO16 (SMS Message Delivery), DUO17 (SMS Message Delivery), DUO18 (SMS Message Delivery), DUO19 (SMS Message Delivery), DUO20 (SMS Message Delivery), DUO21 (SMS Message Delivery), DUO22 (SMS Message Delivery), DUO23 (SMS Message Delivery), DUO24 (SMS Message Delivery), DUO25 (SMS Message Delivery), DUO26 (SMS Message Delivery), DUO27 (SMS Message Delivery), DUO28 (SMS Message Delivery), DUO29 (SMS Message Delivery), DUO30 (SMS Message Delivery), DUO31 (SMS Message Delivery), DUO32 (SMS Message Delivery), DUO33 (SMS Message Delivery), DUO34 (SMS Message Delivery), DUO36 (SMS Message Delivery), DUO37 (SMS Message Delivery), DUO38 (SMS Message Delivery), DUO39 (SMS Message Delivery), DUO40 (SMS Message Delivery), DUO41 (SMS Message Delivery), DUO42 (SMS Message Delivery), DUO43 (SMS Message Delivery), DUO44 (SMS Message Delivery), DUO45 (SMS Message Delivery), DUO46 (SMS Message Delivery), DUO48 (SMS Message Delivery), DUO9 (SMS Message Delivery), DUO49 (SMS Message Delivery), DUO50 (SMS Message Delivery), DUO51 (SMS Message Delivery), DUO52 (SMS Message Delivery), DUO53 (SMS Message Delivery), DUO54 (SMS Message Delivery), DUO55 (SMS Message Delivery), DUO56 (SMS Message Delivery), DUO57 (SMS Message Delivery), DUO58 (SMS Message Delivery), DUO59 (SMS Message Delivery), DUO60 (SMS Message Delivery), DUO61 (SMS Message Delivery), DUO62 (SMS Message Delivery), DUO63 (SMS Message Delivery), DUO64 (SMS Message Delivery), DUO65 (SMS Message Delivery), and DUO35 (SMS Message Delivery).