Increased error rate effecting US-EAST projects on the V3 asset platform

Everything appears to be functioning normally once again. Thanks for your patience while this was resolved.

A small handful of asset transforms continue to fail due to underlying storage issue. We'll be continuing to monitor error rates as service gradually recovers.

In the last 10 minutes, we've seen a noticeable improvement in error rates. A lot of previously-failing asset transforms appear to be successfully processing, however there are still subset that are continuing to fail.

Our asset storage provider has issued an update on their status page:

Recovery operation is currently underway to bring back the impacted systems. This will take between 6 to 12 hours to complete. We will continue to update here as progress is made.

We've received an update from our asset storage provider (Wasabi):

At this time I can say that the initial cooling issue has been resolved by data center operations and the Wasabi Operations team is working to bring all affected server hardware back online as quickly as possible. We do still anticipate a higher-than-normal volume of 500 errors to occur until all hardware is running optimally

We're currently waiting on a further update from our asset storage provider. Error rates don't seem to have varied over the past hour, but we can see some image transforms slowly succeeding. From our observations, there does not appear to be a particular pattern that indicates which transforms will succeed and which ones won't.

No change for the past 30 minutes has been observed, we're continuing to monitor things and will provide an update as soon as we have one.

We've seen a slight improvement in the error rate from our asset storage provider, but a significant number of new asset transforms are still failing. We're continuing to monitor.

Transforms generated prior to this incident seems to be being served fine from our asset CDN. The issue seems to confined to a subset of original asset files.

Our V3 asset platform storage provider has reported an increase in error rate for their US regions. https://status.wasabi.com/incidents/vwt31n6y6sc7 We're seeing this have a knock on effect on projects hosted in our US-EAST clusters that are using the V3 asset platform. We're currently working to mitigate the impact.