All Deployments: SMS delivery issues in the United States
Postmortem
From 8:23 UTC to 14:07 UTC on July 1, 2019, all Duo deployments experienced issues delivering SMS messages to T-Mobile customers with U.S.-based numbers due to a service disruption affecting our third-party telephony providers.
Impacted users were able to use Duo Push, voice call, or other methods during this time to complete authentication. Users asked to verify their number during enrollment could choose a phone call. Users invited to activate Duo Mobile via an SMS link may not have received their invites. Resend activation links to ensure delivery.
The root cause has been identified by Duo’s telephony providers and is now resolved. Duo’s engineering team is committed to reducing the overall impact of similar events going forward by implementing additional proactive monitoring of our telephony providers to reduce failover duration.
Posted Jul 02, 2019 - 12:39 EDT
Resolved
The issue regarding SMS Message Delivery on our deployments is fully resolved and all services are now fully functional.
We will be posting a root-cause analysis (RCA) here once our engineering team has finished its thorough investigation of the issue.
Please make sure to check back or subscribe to be notified when the RCA is posted.
Posted Jul 01, 2019 - 11:08 EDT
Update
Our third party provider is reporting this issue as resolved on their end. We are continuing to monitor an issue causing SMS delivery on all of our deployments to ensure our customers are no longer impacted.
We will follow up to confirm and report on cause and impact.
Posted Jul 01, 2019 - 10:51 EDT
Monitoring
We are currently monitoring an issue causing SMS delivery issues on all of our deployments and are working to correct the issue as soon as possible. Our third-party provider believes this issue is resolved, but we are still investigating the impact and will provide more details as they become available
Please check back here or subscribe to updates for any changes.
Posted Jul 01, 2019 - 10:38 EDT
This incident affected: DUO1 (SMS Message Delivery), DUO2 (SMS Message Delivery), DUO3 (SMS Message Delivery), DUO4 (SMS Message Delivery), DUO5 (SMS Message Delivery), DUO6 (SMS Message Delivery), DUO7 (SMS Message Delivery), DUO8 (SMS Message Delivery), DUO47 (SMS Message Delivery), DUO10 (SMS Message Delivery), DUO11 (SMS Message Delivery), DUO12 (SMS Message Delivery), DUO13 (SMS Message Delivery), DUO14 (SMS Message Delivery), DUO15 (SMS Message Delivery), DUO16 (SMS Message Delivery), DUO17 (SMS Message Delivery), DUO18 (SMS Message Delivery), DUO19 (SMS Message Delivery), DUO20 (SMS Message Delivery), DUO21 (SMS Message Delivery), DUO22 (SMS Message Delivery), DUO23 (SMS Message Delivery), DUO24 (SMS Message Delivery), DUO25 (SMS Message Delivery), DUO26 (SMS Message Delivery), DUO27 (SMS Message Delivery), DUO28 (SMS Message Delivery), DUO29 (SMS Message Delivery), DUO30 (SMS Message Delivery), DUO31 (SMS Message Delivery), DUO32 (SMS Message Delivery), DUO33 (SMS Message Delivery), DUO34 (SMS Message Delivery), DUO36 (SMS Message Delivery), DUO37 (SMS Message Delivery), DUO38 (SMS Message Delivery), DUO39 (SMS Message Delivery), DUO40 (SMS Message Delivery), DUO41 (SMS Message Delivery), DUO42 (SMS Message Delivery), DUO43 (SMS Message Delivery), DUO44 (SMS Message Delivery), DUO45 (SMS Message Delivery), DUO46 (SMS Message Delivery), DUO48 (SMS Message Delivery), DUO9 (SMS Message Delivery), DUO49 (SMS Message Delivery), DUO50 (SMS Message Delivery), DUO51 (SMS Message Delivery), DUO52 (SMS Message Delivery), DUO53 (SMS Message Delivery), DUO54 (SMS Message Delivery), DUO55 (SMS Message Delivery), DUO56 (SMS Message Delivery), DUO57 (SMS Message Delivery), DUO58 (SMS Message Delivery), DUO59 (SMS Message Delivery), DUO60 (SMS Message Delivery), DUO61 (SMS Message Delivery), DUO62 (SMS Message Delivery), DUO63 (SMS Message Delivery), DUO64 (SMS Message Delivery), DUO65 (SMS Message Delivery), and DUO35 (SMS Message Delivery).