Intermittent Traffic Interruption to UK and EU clusters

DigitalOcean has now resolved the incident https://status.digitalocean.com/incidents/q4b09b022nmh.

As things have resumed to normal for a few hours now, we're sufficiently confident the root cause of the issue has been successfully addressed and so are resolving this incident.

DigitalOcean have completed the fix rollout to the UK-LONDON-2 cluster, we'll be monitoring it closely for a little while longer until we're convinced things are stable again.

The EU clusters appear to be returning to normal service after DigitalOcean's fix, we're just waiting on the status of the fix rollout for our UK-LONDON cluster.

Update from DigitalOcean: they have fully rolled out a fix to our impacted EU clusters. The rollout to our UK-LONDON clusters is still in progress.

Response times and timeouts seems to still be fluctuating despite the DigitalOcean's fix rollout.

We're feeding back what we're seeing directly to DigitalOcean's support team to help troubleshoot.

We've received an update from DigitalOcean that they're currently rolling out a fix for this issue.

We've observed a drop in request latency and timeouts across impacted projects.

We're seeing an increase in timeouts happening with our EU and UK clusters.

We're believe it's directly related to this incident that DigitalOcean is experiencing. You can see updates here https://status.digitalocean.com/incidents/q4b09b022nmh

DigitalOcean have raised a status incident https://status.digitalocean.com/incidents/q4b09b022nmh

We'll be closely tracking it and will resolve this incident once they've given the all clear.

We're seeing traffic starting to come through again. We're monitoring things closely to see if things remain stable.

We're working closely with DigitalOcean (the underlying hosting provider for this cluster) to figure out the impact. It appears their entire LON-1 cluster may be down, which is corroborated by reports on social networks.

We're investigating an interruption in traffic to our UK-LONDON-2 cluster.