Incident History

An hypervisor on the Montreal zone is unreachable. One of the FSBucket servers of the zone is hosted on it and is therefore unreachable too. This might impact PHP applications as well as any applications using an FSBucket hosted on this server.

We are awaiting information from our infrastructure provider regarding this incident.

EDIT 19:53 UTC: It seems like multiple servers are impacted at the same time, we believe it to be an issue with a specific OVH rack or room. Multiple services on the zone are thus impacted. We are looking at ways to mitigate the issues.

EDIT 20:05 UTC: The servers are reachable again since a few minutes. We are currently making sure everything is fine. OVH incident can be followed here: https://bare-metal-servers.status-ovhcloud.com/incidents/k664s90jxfj0

EDIT 20:15 UTC: Servers in the impacted rack couldn't reach each other up until now. It could have prevented some services to correctly work. It seems like OVH fixed it before we could report it to them. We continue to making sure everything is working as expected.

EDIT 20:36 UTC: The incident is over. We are redeploying all the applications of the zone to be on the safe side.

We are currently having an ingestion issue on our metrics cluster. The root cause has been identified and we are currently working on a fix. Until this incident is fixed, metrics data points might be missing from your metrics dashboards. Access logs are also impacted but will be re-queued later.

EDIT 14:14 UTC: Metrics ingestion is now back to normal. Access logs are being re-queued and are currently lagging a bit.

EDIT 14:20 UTC: Access logs have been ingested and are now up-to-date. The incident is now over.

EDIT 16:25 UTC: The problem came back, we are working on it.

EDIT 16:56 UTC: The problem is now solved again. Another root cause has been identified and has been fixed.

We are encountering slowness on the Cellar infrastructure. We are investigating why.

EDIT 15:05 UTC: The issue has been found and fixed. Performance went back to normal around 13:45 UTC. Additional measures will be taken to avoid this issue in the future.

Users reported issues while connecting to their database. We are investigating.

09:30 UTC : A huge number of add-ons recently created by malicious users was detected. It was issuing a lot of configuration changes on our reverse proxies, making them unstable.

We banned those users and are watching the situation closely.

MongoDB shared cluster for free addons seems to be under heavy load. We are investigating.

Deployments services are experencing an abnormal load. We have identified the root cause and are fixing it.

12:20 UTC: The deployments are still running slow. We are still cleaning the situation.

13:16 UTC: we have found a deployment loop with the monitoring. We are stopping it…

13:51 UTC: cleaning is done, we are watching to see if deployments are running as expected

14:00 UTC: we have found an abnormal behaviour, we are investigating

D+1 14:30 UTC: we have made a patch for the abnormal behaviour and we are watching deployments

Maintenace is now over

--

Thursday 27 of april at 2:00 PM CEST (12:00AM UTC) we will apply a major update concerning Clever Cloud APIs.
This update prepares some work for future and actual services.

Are you concerned?

All Clever Cloud public regions are concerned. Gov and Private regions aren't concerned, neither On Premise regions.

What's the expected behavior during the maintenance window?

All Applications and Cloud Services will continue to run as expected.
Some APIs calls may be delayed or refused for a few minutes. Deployments may take a bit longer than expected.

We expect services to be fully operational for 3PM CEST.

What do I have to do?

If you're driving your scalability, please anticipate your requirements to be fulfilled by 1PM CEST, since autoscaling won't be as reactive during the maintenance window.
--
We will keep you posted with the process in here and via this twitter thread

The monitoring has detected a burst of connections on regions PAR-SCW (Paris Scaleway) and RBX (Roubaix). Applications may have experienced deconnections and blocked new connections.

EDIT 20:18:00 UTC : The issue is mitigated and we are watching

EDIT 20:50:00 UTC : Everything goes to habitual levels

We are having issues with some deployments. To fix them, we halt deployments for a few minutes.

EDIT 25 of April 08:04 AM UTC: We are still experiencing some deployments issues. Issue have been identified, we are working on a fix.

Some VMs are currently stuck in the STOPPING state. We are investigating.

EDIT 8:00 AM UTC: VMs are no longer stuck.

Reason: a bad user found a way to start a lot of huge instances and run resource-heavy cryptomining operations. It loaded the hypervisors and made some APIs unresponsive. We blocked them and took actions to prevent future abuse of our service.

Metrics and access logs (through Grafana, the Web Console or the API) are experiencing either slowness or missing data. We are currently looking into it.

EDIT 10:40 AM UTC: We have found the issue and we are currently fixing it.

EDIT 04:01 PM UTC: The issue is resolved

Deployments are currently stuck, preventing users from deployment their applications. Other parts of the infrastructure are currently experiencing instabilities.

EDIT 09:35 UTC: The problem should now be mostly resolved. Some services might still have troubles, dedicated incidents will be opened. We continue monitoring the situation.

A maintenance on the storage layer of the metrics platform slowdown the ingestion and we have some log to consume.

EDIT 12:42 UTC: The maintenance operation is complet'ed and no more lag is present

We are currently investigating reverse proxies issues on the Warsaw zone.

EDIT 20:30 UTC: The problem was due to an increased load and capacity has been added to handle it. We continue to monitor the incident.

EDIT 00:53 UTC: We did not see any other issues since 20:30 UTC. This problem is now fixed.

The shared MongoDB cluster is having issues with people abusing it.

We have disabled the creation of new MongoDB DEV plans. This will give us time to setup a new cluster and clean the existing one.

You can still provision the other MongoDB plans.

MongoDB shared cluster is currently down. We are working on getting it back on.

It seems that the cluster got a lot of connections and could not handle the load.

The cluster is currently reconstructing. Waiting for it to finish.

19:40 The cluster has finished reconstructing and is now taking connections.

We've had reverse proxies instabilities on the Warsaw zone between 16:15 UTC and 16:22 UTC. During that time, some connections might have been refused or closed unexpectedly. The problem has been fixed and the underlying issue has been found.

Cellar control plane has detected inconsistencies. We are investigating the issue.

EDIT 09:25 UTC : We begin the recovrey process. We are waiting for the process to terminate

EDIT 09:32 UTC : The recovery process has ended successfully cluster is healthy

An hypervisor has crashed

We are observing that few deployments are freeze or unsync with load balancing system.

EDIT 15:00 UTC : Deployment system is now in-sync and freeze deployement are up and running