Hello Blackthorn Customers - We would like to apologize for the caching issue, which was resolved this morning. We understand for many of you this may have had a significant business impact if you were attempting to update in-flight events with heavy traffic, or needed to delay a new event launch.

We believe this began at the beginning of yesterday (the inability to update a live event). Events did not go down, just the ability to update them wasn’t available. After further review of the root cause with our engineering team, we wanted to send over a clarification to the email sent earlier today.

Our platform was allocating insufficient resources to our caching infrastructure (on AWS). The monitoring of this particular system resource was not specific enough to pick up on this overflow. Lastly we did not size the resource for caching properly and at the time, did not have a notification system in place to indicate that traffic was not processing even though the workers (system processes) were live.

To address this moving forward, we’ve increased the available resources and are enhancing our ability to more granularly monitor production resources. If you run into any issues or questions going forward, please reach out to our support team here: https://community.blackthorn.io/s/support

Sincerely,
Blackthorn Team