Intermittent Traffic Interruption to UK and EU clusters

Resolved
Resolved

DigitalOcean has now resolved the incident https://status.digitalocean.com/incidents/q4b09b022nmh.

As things have resumed to normal for a few hours now, we're sufficiently confident the root cause of the issue has been successfully addressed and so are resolving this incident.

Avatar for
Recovering

DigitalOcean have completed the fix rollout to the UK-LONDON-2 cluster, we'll be monitoring it closely for a little while longer until we're convinced things are stable again.

Avatar for
Recovering

The EU clusters appear to be returning to normal service after DigitalOcean's fix, we're just waiting on the status of the fix rollout for our UK-LONDON cluster.

Avatar for
Identified

Update from DigitalOcean: they have fully rolled out a fix to our impacted EU clusters. The rollout to our UK-LONDON clusters is still in progress.

Avatar for
Identified

Response times and timeouts seems to still be fluctuating despite the DigitalOcean's fix rollout.

We're feeding back what we're seeing directly to DigitalOcean's support team to help troubleshoot.

Avatar for
Recovering

We've received an update from DigitalOcean that they're currently rolling out a fix for this issue.

We've observed a drop in request latency and timeouts across impacted projects.

Avatar for
Identified

We're seeing an increase in timeouts happening with our EU and UK clusters.

We're believe it's directly related to this incident that DigitalOcean is experiencing. You can see updates here https://status.digitalocean.com/incidents/q4b09b022nmh

Avatar for
Recovering

DigitalOcean have raised a status incident https://status.digitalocean.com/incidents/q4b09b022nmh

We'll be closely tracking it and will resolve this incident once they've given the all clear.

Avatar for
Recovering

We're seeing traffic starting to come through again. We're monitoring things closely to see if things remain stable.

Avatar for
Investigating

We're working closely with DigitalOcean (the underlying hosting provider for this cluster) to figure out the impact. It appears their entire LON-1 cluster may be down, which is corroborated by reports on social networks.

Avatar for
Investigating

We're investigating an interruption in traffic to our UK-LONDON-2 cluster.

Avatar for
Began at:

Affected components
  • Dashboard
  • Task Runner
  • Clusters
    • EU-WEST-1
    • EU-WEST-2
    • EU-WEST-3
    • EU-WEST-5
    • UK-LONDON-2
  • servd.host Website