Incident History

This maintenance concerns the migration of our cellar-c1 Cellar cluster. Affected customers have been emailed multiple times since the January regarding this service end of life.

As a reminder, the service will be shut down on 21/03/22. A few network brownouts will be applied to remind customers that they need to migrate their data.

A total of 5 brownouts will be applied. During these planned downtime, the service will refuse any connections, be it HTTP or HTTPS.

This brownout will happen on 18/03/22 10:00 UTC for a 30 minutes window

Our support team stays at your disposal for any questions.

EDIT 11:00 UTC: The brownout has started and will last for 30 minutes.

EDIT 11:30 UTC: The brownout has ended. The service will be decommissioned next Monday.

This maintenance concerns the migration of our cellar-c1 Cellar cluster. Affected customers have been emailed multiple times since the January regarding this service end of life.

As a reminder, the service will be shut down on 21/03/22. A few network brownouts will be applied to remind customers that they need to migrate their data.

A total of 5 brownouts will be applied. During these planned downtime, the service will refuse any connections, be it HTTP or HTTPS.

This brownout will happen on 14/03/22 09:30 UTC for a 30 minutes window.

Our support team stays at your disposal for any questions.

EDIT 09:36 UTC: The brownout is starting. It will last for 30 minutes.

EDIT 10:07 UTC: The brownout has ended. Next one will happen on 16/03/22 16:00 UTC for a 30 minutes window.

This maintenance concerns the migration of our cellar-c1 Cellar cluster. Affected customers have been emailed multiple times since the January regarding this service end of life.

As a reminder, the service will be shut down on 21/03/22. A few network brownouts will be applied to remind customers that they need to migrate their data.

A total of 5 brownouts will be applied. During these planned downtime, the service will refuse any connections, be it HTTP or HTTPS.

This brownout will happen on 11/03/22 14:00 UTC for a 10 minutes window.

Our support team stays at your disposal for any questions.

EDIT 14:00 UTC: The brownout is starting and will last for 10 minutes.

EDIT 14:10 UTC: The brownout has ended. Next one will happen on 14/03/22 09:30 UTC for a 30 minutes window.

This maintenance concerns the migration of our cellar-c1 Cellar cluster. Affected customers have been emailed multiple times since the January regarding this service end of life.

As a reminder, the service will be shut down on 21/03/22. A few network brownouts will be applied to remind customers that they need to migrate their data.

A total of 5 brownouts will be applied. During these planned downtime, the service will refuse any connections, be it HTTP or HTTPS.

This brownout will happen on 09/03/22 10:00 UTC for a 10 minutes window.

Our support team stays at your disposal for any questions.

EDIT 10:00 UTC: The brownout has started.

EDIT 10:10 UTC: The brownout has ended. Next one will happen on 11/03/22 14:00 UTC

Our Cellar C1 cluster service has currently connectivity issues leading to failed requests. We are investigating with our network provider the reason of those issues.

Edit: Connectivity issues has been solved by our network provider. The service should run as expected

We identified issues on our metrics/accesslogs storage. We are working to fix the problem which is currently causing timeouts on queries.

EDIT 10:27 UTC: Queries have returned to normal, Metrics and Access logs should now be reachable. We are monitoring the queries.

EDIT 11:03 UTC: Queries have returned to normal, Metrics and Access logs should now be reachable.

An add-on reverse proxy started behaving erratically. This triggered timeouts and unreachability for some add-ons if the active connections were proxied through it. It has been restarted, which fixed the issue.

Sorry for the inconvenience.

Some of the databases hosted on that cluster were unreachable due to a node failure during a few hours. The problem has been fixed and the failure will be investigated further. Dedicated databases were not impacted.

We are currently having connections issues toward Scaleway infrastructure from one of our datacenters in Paris. We are investigating this issue with our network providers.

EDIT 23:15 UTC: The connectivity is now back since 23:07 UTC with our network provider saying that the issue has been resolved. This incident is now closed on our end. Sorry for the inconvenience.

Our Cellar C1 cluster service is currently unreachable by our Paris infrastructure, leading to failed requests. This cluster is the old cluster, with either domains cellar-c1.clvrcld.net or cellar.services.clever-cloud.com. We are investigating the issue.

EDIT 22:42 UTC: After a quick investigation, only one of the 3 IP that is serving those domains is having troubles reaching other nodes of the cluster. The IP has been dropped from the DNS. Meanwhile, we are investigating the issue with our network provider.

EDIT 22:39 UTC: Lowering the severity to Performance Issues. Ticket has been open with our network provider.

EDIT 23:15 UTC: The connectivity is now back since 23:07 UTC with our network provider saying that the issue has been resolved. We will wait a bit before adding back the IP of the faulty node in the DNS just to be sure but this incident is now closed on our end. Sorry for the inconvenience.

Generation of certificates for newly added domain is currently delayed due to a rate limit issue. A fix has been issued on our end and the situation should come back to normal in a few hours.

EDIT 19:16 UTC+1: This does not impact renewal of certificates.

EDIT 19:36 UTC+1: We are now under the rate limit, newly added domains should have their certificates generated in a few minutes, as usual. Sorry for the inconvenience.

We need to perform a maintenance operation on the free PostgreSQL shared cluster of the Paris zone. A fail-over will be initiated and applications may have troubles connecting to the new leader. Make sure to restart them if needed.

The fail-over will be done in the upcoming hour.

EDIT 15:17 UTC: The cluster will be fail-over in the next few minutes. Some queries might be failing as soon as the leader goes down and until your application correctly connect to the new leader.

EDIT 15:28 UTC: The fail-over has been done. Make sure to restart your applications if they can't connect to their add-on.

We are currently having difficulties with the logs pipeline. This impacts live logs in the Console / CLI as well as drain logs. We are working on it.

EDIT 14:25 UTC: The issue has been fixed. A fix has been scheduled for deployment this afternoon which should reduce those delivery issues events. We will monitor the fix closely once it gets deployed.

We are currently having difficulties with the logs pipeline. This impacts live logs in the Console / CLI as well as drain logs. We are working on it.

EDIT 15:07 UTC: Live logs and drains are back. Some drains logs may have been lost during the recovery process. Sorry for the inconvenience.

EDIT 15:52 UTC: Live logs and drains are down again, we are looking into it

We identified issues on our metrics/accesslogs storage. We are working to fix the problem which is currently causing timeouts on queries.

EDIT 15:52 UTC: Queries have returned to normal, Metrics and Access logs should now be reachable.

Metrics and access logs are currently having an ingestion issue. Some metrics points will be lost, access logs will be kept and ingested at a later time.

Ingestion is now starting at full capacity again. There will be some delay before having up-to-date access logs but it should be good in a few hours. Sorry for the inconvenience.

The log drains infrastructure went down last night (2022-02-14 around 5 AM) and some drains were lost / are broken.

We are still identifying which ones are broken to restart them. If you see that your drains are broken, please contact the support so we can restart them!

Edit 15:11 — we restarted all drains to be sure. Edit 16:27 — Most of the drains are still broken. We are trying to fix the issue by deleting and re-creating message queues in the logs infrastructure. Edit 16:37 — Deleting and creating back everything seems to have cleaned up the situation. Drains seem to be working again!

Logs and logs drains are experiencing issues. We are investigating.

EDIT 20:32 UTC - fixed.

We identified issues on our metrics/accesslogs storage. We are working to fix the problem which is currently causing timeouts on queries.

EDIT 17:45 UTC: The incident is over, sorry for the inconvenience.

We are experiencing issues with logs collection and distribution (drains included).

EDIT 20:27 UTC: We identified the issue, and the resolution is on going.

EDIT 20:54 UTC : Fixed.