We’re closing this incident as resolved. No action is needed on your end as the patch was deployed for our messaging API service.
Root Cause Analysis: An optional parameter for our messaging API was sometimes getting passed with an empty value. This caused the intermittent issue where some messages would fail to send. To fix this we deployed a hotfix patch and are now only passing the parameter if it actually has a value.
There was recently a small API change for the messaging service provider we utilize for sending messaging. They believe this caused a validation rule to fail when we passed the parameter with an empty value. Moving forward we will run our regression tests more frequently to proactively catch these sorts of issues. We will also review our integration with this provider to ensure we are only passing parameters with valid values.
Posted Feb 15, 2023 - 22:11 UTC
Monitoring
Our team deployed a hotfix patch this morning around 9 am EST (2 pm UTC) to resolve this issue. Over the past 2 hours, we have not seen any more errors. As such, we’re updating the status of this incident to “Monitoring”.
No action needs to be taken from your end. Please reach out if your messaging application is still experiencing this intermittent deliverability issue. We will continue to monitor this incident for the rest of the day before we officially close it out.
Posted Feb 15, 2023 - 16:22 UTC
Update
Our teams are continuing to investigate this API issue. If your Messaging application is experiencing this intermittent deliverability issue, we recommend granting Subscriber access to the environment for our teams to continue their investigation. We are working as quickly as possible to resolve the issue.
Posted Feb 15, 2023 - 01:48 UTC
Investigating
The Engineering team and Messaging Platform team are currently investigating an issue relating to intermittent message deliverability. We will provide updates here as the investigation continues. Thank you for your patience. We are urgently working to restore SMS deliverability.