The "Warning: Your checkout failed, please try again." error that Attendees were intermittently seeing when trying to register via Event checkout has now been resolved. Our engineers are monitoring in case any further issues arise.
This error was occurring because of a previous internal change to how checkouts were routing related to registration volume. This fix worked in the short term in accomodating more volume, but caused this new issue with checkouts. The Blackthorn cache was making sessions not ‘sticky’ and customers were jumping instances, which was breaking sessions and resulting some checkouts going through and not others. A long term fix has been implemented and we will continue to monitor the results and make improvements. If you have any questions please reach out via our support page at https://community.blackthorn.io/s/support.
Posted Mar 11, 2022 - 15:52 UTC
Update
After further investistgation, Blackthorn Event pages were down due to unusually high levels of traffic, which was caused by demand increasing faster than infrastructure. This has now been resolved and Event Pages should render without any errors.
Please note there is still an intermittent issue where attendees may encounter the following error when trying to register: "Warning: Your checkout failed, please try again." This is very intermittent and our engineering team is working on addressing this as soon as possible.
Please find action items taken by our engineering team below: - We are setting up Cloudflare Load Balancing - We upgraded one of our Heroku instances to the single largest capacity that Heroku has - We are confirming the ‘worker’ is working right - routing traffic to different Heroku instances based upon geography then splitting in the US - Long-term we are migrating to AWS which allows us to horizontally scale better preventing future outages like this
We are currently monitoring the situation but please reach out to Blackthorn Support at https://community.blackthorn.io/s/support if you run into any issues going forward.
Posted Mar 10, 2022 - 21:11 UTC
Monitoring
Our team is actively investigating this issue and we will provide regular updates as more information becomes available.
Posted Mar 10, 2022 - 19:07 UTC
Identified
We’ve been experiencing intermittent errors with Event Pages and AttendeeLinks this morning and our developers are closely monitoring app traffic. Just like all other Salesforce end users, we are also subject to Salesforce API limits, which govern available resources to ensure stable infrastructure performance for all. Unusually high traffic coupled with the known limitations of the Salesforce platform cause the intermittent 400 error. While we can’t change the governing limits of Salesforce, we have some infrastructure changes to our apps being prioritized over the next few releases to compensate for those Salesforce limits and improve performance.