All Deployments: SMS Delivery Issues for some Canadian Customers
Incident Report for Duo
Postmortem

Degraded SMS Delivery to Canadian Area Codes - All Deployments

Incident Report - 2019/09/12

Summary

From 9:51 a.m. EDT to 1:09 p.m. EDT on August 30, 2019, SMS messages containing Duo activation links or authentication passcodes were not reaching users of three cellular carriers in Canada.

Details

Duo’s Support Team received notification at 9:51 a.m of SMS delivery issues affecting certain customers. After initial investigation, the Support team alerted Duo’s Engineering team at 10:31 a.m. EDT with confirmation that some end users were not receiving SMS messages. At 11:30 a.m. it was determined that the primary Duo telephony service was not delivering messages to users with Canadian area codes.

Following investigation, the Engineering Team decided to failover to a secondary telephony provider. Duo Engineering began deploying the changes across production at 12:04 p.m. By 12:37 p.m., we received reports from affected customers that SMS messages were being delivered, and by 1:09 p.m. the deployment to all customers was complete. Following the solution’s deployment, we monitored the delivery of messages and confirmed the service was restored.

After analysis, it was determined that the primary telephony provider was unable to communicate with several carriers in Canada. The Duo Engineering Team is exploring methods to more quickly route traffic to alternate providers if needed in the future. We are also investigating how to detect carrier issues to allow for faster failover.

Posted Sep 13, 2019 - 11:00 EDT

Resolved
We have confirmed that the issues with SMS delivery to some Canadian customers are fully resolved.
Posted Sep 12, 2019 - 16:54 EDT
Monitoring
Our Engineering team has successfully resolved issues with intermittent SMS delivery to our Canadian customers, and we have confirmed that affected customers are seeing successful delivery.
We will continue to monitor the issue to ensure it is completely resolved.
Posted Sep 12, 2019 - 13:13 EDT
Update
We are continuing to investigate this issue.
Posted Sep 12, 2019 - 12:02 EDT
Investigating
We are currently investigating an issue causing SMS delivery issues for some Canadian customers and are working to correct the issue as soon as possible.

Please check back here or subscribe to updates for any changes.
Posted Sep 12, 2019 - 12:00 EDT
This incident affected: DUO1 (SMS Message Delivery), DUO2 (SMS Message Delivery), DUO3 (SMS Message Delivery), DUO4 (SMS Message Delivery), DUO5 (SMS Message Delivery), DUO6 (SMS Message Delivery), DUO7 (SMS Message Delivery), DUO8 (SMS Message Delivery), DUO47 (SMS Message Delivery), DUO10 (SMS Message Delivery), DUO11 (SMS Message Delivery), DUO12 (SMS Message Delivery), DUO13 (SMS Message Delivery), DUO14 (SMS Message Delivery), DUO15 (SMS Message Delivery), DUO16 (SMS Message Delivery), DUO17 (SMS Message Delivery), DUO18 (SMS Message Delivery), DUO19 (SMS Message Delivery), DUO20 (SMS Message Delivery), DUO21 (SMS Message Delivery), DUO22 (SMS Message Delivery), DUO23 (SMS Message Delivery), DUO24 (SMS Message Delivery), DUO25 (SMS Message Delivery), DUO26 (SMS Message Delivery), DUO27 (SMS Message Delivery), DUO28 (SMS Message Delivery), DUO29 (SMS Message Delivery), DUO30 (SMS Message Delivery), DUO31 (SMS Message Delivery), DUO32 (SMS Message Delivery), DUO33 (SMS Message Delivery), DUO34 (SMS Message Delivery), DUO36 (SMS Message Delivery), DUO37 (SMS Message Delivery), DUO38 (SMS Message Delivery), DUO39 (SMS Message Delivery), DUO40 (SMS Message Delivery), DUO41 (SMS Message Delivery), DUO42 (SMS Message Delivery), DUO43 (SMS Message Delivery), DUO44 (SMS Message Delivery), DUO45 (SMS Message Delivery), DUO46 (SMS Message Delivery), DUO48 (SMS Message Delivery), DUO9 (SMS Message Delivery), DUO49 (SMS Message Delivery), DUO50 (SMS Message Delivery), DUO51 (SMS Message Delivery), DUO52 (SMS Message Delivery), DUO53 (SMS Message Delivery), DUO54 (SMS Message Delivery), DUO55 (SMS Message Delivery), DUO56 (SMS Message Delivery), DUO57 (SMS Message Delivery), DUO58 (SMS Message Delivery), DUO59 (SMS Message Delivery), DUO60 (SMS Message Delivery), DUO61 (SMS Message Delivery), DUO62 (SMS Message Delivery), DUO63 (SMS Message Delivery), DUO64 (SMS Message Delivery), DUO65 (SMS Message Delivery), and DUO35 (SMS Message Delivery).