Incident History

A human error caused a configuration error on all public PAR reverse proxies which prevented them from reloading their configuration from 09:20:35 UTC to 09:24:40 UTC.

An automatic restart at 09:21:48 UTC made them unavailable until the configuration was re-generated without the error at 09:24:40 UTC.

Steps will be taken to prevent this error from happening again.

We are suffering network loss on some of our servers in one of our datacenters. We are currently aware of the root cause and working on it.

EDIT 22:49 UTC: Our API is also down for now, that's expected. The console is therefore down too. Clients websites remain accessible.

EDIT 23:11 UTC: Network came back 5 minutes ago, we are currently checking if everything is ok

EDIT 23:26 UTC: Applications with fs-bucket (including PHP applications) may have issues loading because their connection to the fs-bucket server, if this server was on the datacenter who lost the connection.

EDIT 00:26 UTC: Applications with fs-buckets are currently redeploying. Most of them successfully reconnected (sometimes after several minutes) to their bucket server. The incident is over.

Deployments will be disabled for up to 15 minutes on Thursday 2019-06-27 at 19:00 UTC (21:00 CEST).

We will perform a migration of the Git repositories. Once deployments are enabled again, you may have to wait a few more minutes depending on your DNS cache.

19:00 UTC: Maintenance is starting, deployments are now disabled (except for Github deployments).

19:13 UTC: The maintenance will last longer than initially planned, we are experiencing an issue and are looking into it.

19:15 UTC: The issue is fixed. We are making sure that everything is indeed fine. Some deployments may now go through, depending on your DNS cache.

19:30 UTC: Maintenance is over; if you encounter an issue, please refresh your DNS cache.

Deployments will be disabled for up to 15 minutes on Thursday 2019-06-27 at 10:00 UTC (12:00 CEST).

We will perform a migration of the Git repositories. Once deployments are enabled again, you may have to wait a few more minutes depending on your DNS cache.

EDIT: This has been postponed.

Deployments will be disabled for up to an hour on Thursday 2019-06-20 starting at 10:00 UTC (12:00 CEST).

It should be quicker than that but if you do have deployments planned, make sure to start them well before the beginning of the maintenance.

EDIT 10:01 UTC: Maintenance is starting now, deployments are disabled.

EDIT 10:19 UTC: Deployments are enabled again.

EDIT 10:31 UTC: Deployments are disabled again. Dedicated reverse proxies for Clever Cloud APIs are out of sync, our APIs are down at the moment. We are working on it.

EDIT 10:39 UTC: Main API is back online.

EDIT 10:47 UTC: Reverse proxies are in sync, deployments are enabled again. We are cleaning up.

EDIT 10:53 UTC: Maintenance is over.

This cluster is experiencing elevated error rates and response times. It is currently a bit overloaded following the restart of a few nodes which crashed.

It should go back to normal in 30 to 60 minutes.

EDIT 8:56 UTC: There are still clean-up operations in progress which slow down the cluster. Error rate is going down though.

EDIT 9:55 UTC: Incident over since 9:40

Some cleverapps.io domains are experiencing time outs, we are investigating the issue

EDIT 18:41 UTC: The problem should now be fixed since a couple of minutes. We gathered information as to why this problem happened and will try to narrow it down.

A human error triggered a lot of false positives regarding applications status. This in turn queued hundreds of automatic deployments.

The issue is now fixed, but deployments will take a little while longer to start until the queue is consumed.

EDIT: Incident over at 09:40 UTC

A node from the old Cellar cluster restarted at 21:30 UTC. While it went okay at first thanks to the restart of the few nodes a few days ago, it started emitting HTTP 500 errors or timeouts, as it was before. Service should be back online in a few hours once the cluster stabilized itself again.

The new Cellar cluster is not impacted by those issues.

EDIT 23:40 UTC: Cluster now seems to be in a good shape again

Some deployments seem to keep building (even if the build succeeds, another build starts). We are looking into it.

EDIT 12:33 UTC: We may have identified the root cause. It may be due to a change that happened this morning. We will revert it.

EDIT 12:43 UTC: The change has been reverted and we confirm that it resolves the issue. Sorry for the inconvenience.

One node of our old Cellar cluster is restarting, some requests are failing (timeouts or 500 errors). This will be resolved once the node has fully restarted. We may need to restart more nodes right after.

EDIT 23:30 UTC: Other nodes need to be restarted. We saw <1% of failing requests, expect the same amount for the remaining restarts.

EDIT 02:00 UTC: Nodes have been restarted, failing requests are getting lower and lower, still under 1%.

Our main API is currently having some troubles to respond to requests in a timely manner. We are investigating it.

EDIT 15:32 UTC: The issue has been identified, we are currently re-deploying the API. Console is still unavailable.

EDIT 15:34 UTC: The API successfully redeployed and is now available. Console is now available too. The incident is over.

We will restart a part of our logs system, it will take 2 minutes. After this interruption, the logs produced during the restart will be available but the logs ordering will be lost. This restart is a part of our new logs system development.

EDIT 14:27UTC: finished.

Two Factor Authentication is currently down, resulting in users unable to log in.

Cellar c1 may have issues. Some nodes do not restart correctly.

Logs ingestion is currently having issue. We start investigating the issue

EDIT 15:31 UTC: The issue has been fixed. Some of the logs were lost but not all of them, you should have the last ~15 minutes, the buffer wasn't large enough to keep them all. We will increase it next week.

One of our MySQL shared cluster is under high load. We are investigating.

EDIT 16:20 UTC: Problematic queries have been killed and the cluster load is going down. We continue to monitor the situation but it should go back to normal. We also have a newer MySQL shared cluster on MySQL version 8. You can migrate your database to it using the "Migrate" tool.

EDIT 16:45 UTC: The performance issue is back, we are trying to narrow down the issue

EDIT 17:00 UTC: Performances are again back to normal. We will keep an eye on it. Meanwhile, do not hesitate to migrate to our new cluster to avoid this issue.

EDIT 10/05/19 08:10 UTC: The issue has come back.

EDIT 10/05/19 12:00 UTC: Owners of the potential abusive queries have been notified. Cluster performances are back to normal. As usual, we will keep an eye on it.

We have an issue in the logs ingestion pipeline. We are working on it.

EDIT: Issue resolved at 15:48:20 UTC

We are recording important error rates on Cellar between 500 and 503 errors.

EDIT 09:41 UTC: 503 errors are now gone but were replaced by 500 errors that get triggered after a few seconds. We are checking the cluster's state

EDIT 10:10 UTC: Error rate is decreasing but continue to be important. Deployments are also impacted by this issue if you are using build cache.

EDIT 10:27 UTC: Error rate is still at ~20% and continue to decrease.

EDIT 11:52 UTC: We did not receive any errors since 11:40 UTC, the cluster is now in good shape and everything should be back to normal.

This cellar cluster will soon be deprecated (new cellar add-ons are already created on an up-to-date cluster) in favor of a better and maintained version.

Clever Cloud API is very slow, we are investigating.

EDIT 16:11 UTC: fixed.