Incident History

For security reasons, we will migrate our DEV MySQL services on the Paris (PAR) region. Applications using those services will be impacted.

Only the par dev cluster will be updated during this maintenance.

The maintenance will take place on Monday 23rd of October 2023, between 11:45 UTC+2 and 15:00 UTC+2.

For the entire duration of the update, the services will not be available.

The time required to perform the update is estimated between 1 and 2 hours. However, total downtime might be longer as every application using the cluster will need to be restarted.

In case you have connection issues after those updates, you can manually trigger a redeployment of your linked applications.

If you do not want to be impacted by your DEV addon being offline, you can still order or migrate to a dedicated one before this maintenance starts.

Our support team is available for any questions via the ticket center in the console.

EDIT 2023-10-23 11:50 UTC+2: Maintenance is starting.

EDIT 2023-10-23 12:20 UTC+2: Dev addons are now available again. We will restart linked applications

EDIT 2023-10-23 12:22 UTC+2: We investigate an error while creating new DEV addons

EDIT 2023-10-23 12:40 UTC+2: New DEV addons can now be created. All applications linked to DEV addons are currently restarting

EDIT 2023-10-23 13:00 UTC+2: All applications linked to DEV addons have restarted

A maintenance on our notification stack is scheduled for this afternoon. This means services such as WebHooks and Email Notifications will not work.

Downtime is expected to last between 30 minutes to 1 hour.

[14:15 UTC] All notifications services are now up and running

This night, we had network connectivity issues when adding new bgp peer. Those connectivity issues has resulted in desynchronisation of our load balancers. You may have experienced connectivity issue to your database and unavailability of your applications.

PHP+FTP applications may have seen degraded performances and / or timeouts those past few hours on the Paris region due to a configuration issue of the underlying disk.

The configuration has been fixed at 21:00 UTC and disk access time are now in the normal range. We will keep monitoring the situation in the upcoming days to make sure performance stays in normal ranges.

Our security updates will need us to update nodes from our DEV mongodb cluster on PAR.

If you use the mongodb uri correctly, it should only disrupt your application for a few seconds. Otherwise, expect up to two hours of maintenance.

We are encountering issues with our log systems.

The fetch of logs can take a while.

EDIT 21:37 UTC - fixed.

For security reasons, we will update our logs collection systems.

The logs collection (logs drains too) will be unavailable during the maintenance.

EDIT 00:00 UTC: The maintenance is now over.

For security reasons, we will update the our FS Bucket services on the Paris (PAR) region. Applications using those services will be impacted.

FS Bucket hosts that will be updated during this maintenance are: n19 and n20.

The maintenance will take place on Friday 20 October 2023, between 12:00 UTC+2 and 14:00 UTC+2.

During the update of each server host, the services will only be available in read-only mode. Once the update is complete, linked applications will be restarted automatically to take into account the environment variables of the updated services and to restore write capacity.

The required update time is estimated at 1 hour but the total time until the applications are restarted might be longer.

In case you have write issues after those updates, you can manually initiate a redeployment of your linked applications in order to avoid waiting for the automatic redeployment.

To check if your services are impacted, you can consult your FS Bucket’s server in the Dashboard tab of your add-ons, in the “Cluster information” section and thus determine the update day(s) that concerns you.

Specific case - old applications

Please check if you have any old applications (>5 years) that are still using a buckets.json file in their code repository, as we will not be able to prioritize the redeployment of these applications and they will most likely suffer from read-only FS Bucket for an extended time. We therefore recommend that you now mount FS Bucket by environment variable (ideally by linking the add-on to your application). See more details in this documentation page: https://www.clever-cloud.com/doc/deploy/addon/fs-bucket/#configuring-your-application

Our support is available for any questions via the ticket center in the console.

EDIT 2023-10-20 12:00 UTC+2: Maintenance is starting.

EDIT 2023-10-20 13:24 UTC+2: Applications are currently redeploying.

EDIT 2023-10-20 14:20 UTC+2: Applications have redeployed. We are cleaning things up.

EDIT 2023-10-20 16:12 UTC+2: The maintenance is over.

For security reasons, we will update the our FS Bucket services on the Paris (PAR) region. Applications using those services will be impacted.

FS Bucket hosts that will be updated during this maintenance are: n10 and n17.

The maintenance will take place on Thursday 19 October 2023, between 12:00 UTC+2 and 14:00 UTC+2.

During the update of each server host, the services will only be available in read-only mode. Once the update is complete, linked applications will be restarted automatically to take into account the environment variables of the updated services and to restore write capacity.

The required update time is estimated at 1 hour but the total time until the applications are restarted might be longer.

In case you have write issues after those updates, you can manually initiate a redeployment of your linked applications in order to avoid waiting for the automatic redeployment.

To check if your services are impacted, you can consult your FS Bucket’s server in the Dashboard tab of your add-ons, in the “Cluster information” section and thus determine the update day(s) that concerns you.

Specific case - old applications

Please check if you have any old applications (>5 years) that are still using a buckets.json file in their code repository, as we will not be able to prioritize the redeployment of these applications and they will most likely suffer from read-only FS Bucket for an extended time. We therefore recommend that you now mount FS Bucket by environment variable (ideally by linking the add-on to your application). See more details in this documentation page: https://www.clever-cloud.com/doc/deploy/addon/fs-bucket/#configuring-your-application

Our support is available for any questions via the ticket center in the console.

For security reasons, we will update the our FS Bucket services on the Paris (PAR) region. Applications using those services will be impacted.

FS Bucket hosts that will be updated during this maintenance are: n15 and n16.

The maintenance will take place on Wednesday 18 October 2023, between 12:00 UTC+2 and 14:00 UTC+2.

During the update of each server host, the services will only be available in read-only mode. Once the update is complete, linked applications will be restarted automatically to take into account the environment variables of the updated services and to restore write capacity.

The required update time is estimated at 1 hour but the total time until the applications are restarted might be longer.

In case you have write issues after those updates, you can manually initiate a redeployment of your linked applications in order to avoid waiting for the automatic redeployment.

To check if your services are impacted, you can consult your FS Bucket’s server in the Dashboard tab of your add-ons, in the “Cluster information” section and thus determine the update day(s) that concerns you.

Specific case - old applications

Please check if you have any old applications (>5 years) that are still using a buckets.json file in their code repository, as we will not be able to prioritize the redeployment of these applications and they will most likely suffer from read-only FS Bucket for an extended time. We therefore recommend that you now mount FS Bucket by environment variable (ideally by linking the add-on to your application). See more details in this documentation page: https://www.clever-cloud.com/doc/deploy/addon/fs-bucket/#configuring-your-application

Our support is available for any questions via the ticket center in the console.

For security reasons, we will update the our FS Bucket services on the Paris (PAR) region. Applications using those services will be impacted.

FS Bucket hosts that will be updated during this maintenance are: n12 and n13.

The maintenance will take place on Tuesday 17 October 2023, between 12:00 UTC+2 and 14:00 UTC+2.

During the update of each server host, the services will only be available in read-only mode. Once the update is complete, linked applications will be restarted automatically to take into account the environment variables of the updated services and to restore write capacity.

The required update time is estimated at 1 hour but the total time until the applications are restarted might be longer.

In case you have write issues after those updates, you can manually initiate a redeployment of your linked applications in order to avoid waiting for the automatic redeployment.

To check if your services are impacted, you can consult your FS Bucket’s server in the Dashboard tab of your add-ons, in the “Cluster information” section and thus determine the update day(s) that concerns you.

Specific case - old applications

Please check if you have any old applications (>5 years) that are still using a buckets.json file in their code repository, as we will not be able to prioritize the redeployment of these applications and they will most likely suffer from read-only FS Bucket for an extended time. We therefore recommend that you now mount FS Bucket by environment variable (ideally by linking the add-on to your application). See more details in this documentation page: https://www.clever-cloud.com/doc/deploy/addon/fs-bucket/#configuring-your-application

Our support is available for any questions via the ticket center in the console.

EDIT 2023-10-17 12:10 UTC+2: The maintenance is starting. FSBucket servers are set in read-only mode.

EDIT 2023-10-17 12:47 UTC+2: Applications are being redeployed to use the new FSBucket server. You can also start a deployment on your side to speed things up.

EDIT 2023-10-17 16:15 UTC+2: The maintenance is over. All applications should now have access to their fsbucket since 14:00 UTC+2. Please reach out to our support team if you have any issues following this maintenance.

For security reasons, we will update the PHP FTP services on the Paris (PAR) region used by PHP+FTP applications. PHP applications using the Git deployment method are not impacted by this maintenance.

PHP FTP hosts that will be updated during this maintenance are: n11 and n18.

The maintenance will take place on Monday 16 October 2023, between 12:00 UTC+2 and 14:00 UTC+2.

During the update of each server host, the services will only be available in read-only mode. Once the update is complete, linked applications will be restarted automatically to take into account the environment variables of the updated services and to restore write capacity.

The required update time is estimated at 1 hour but the total time until the applications are restarted might be longer.

In case you have write issues after those updates, you can manually initiate a redeployment of your PHP+FTP applications in order to avoid waiting for the automatic redeployment.

Our support is available for any questions via the ticket center in the console. This maintenance will be updated during the maintenance window.

EDIT 2023-10-16 12:03 UTC+2: The maintenance will begin shortly. FSBucket add-on hosted on those servers will soon become read-only.

EDIT 2023-10-16 12:09 UTC+2: FSBuckets are now read-only

EDIT 2023-10-16 12:54 UTC+2: Applications are being redeployed to use the new FSBucket server. You can also start a deployment on your side to speed things up.

EDIT 2023-10-16 14:04 UTC+2: The maintenance is over. All applications should now have access to their fsbucket since 13:30 UTC+2. Please reach out to our support team if you have any issues following this maintenance.

An hypervisor had its internal network unreachable between 15:16 and 15:20 UTC. During that time, services on the hypervisors may have suffered a total loss of network connectivity. Applications have been redeployed because of the unreachability. The issue has been fixed and we are monitoring the situation.

EDIT 15:49 UTC: All services are now reachable again, the incident is now over.

We are detecting a data lag on our Metrics / AccessLog stack. You may not experience the latest datapoints. We are working on it.

EDIT 10:00 PM: lag has been fully absorbed

One hypervisor is unreachable. We are investigating it.

It may impact some databases that are hosted on top of this hypervisor.

06:20 The hypervisor seems to have encountered a kernel panic. It has been rebooted and we fixed the kernel version to avoid future Kernel Panic.

06:45 Now that the hypervisor is back up, we are cleaning the situation: checking all add-ons instances have rebooted successfully, that all applications have redeployed successfully.

07:21 Everything is now back to normal

Starting 2023-10-06 23:00 UTC, we will conduct a maintenance operation on one of the key parts of the deployment system. With the security margins, we expect it to take 2 hours.

As a result, deployments on all our zones will be disabled between 11:00 and 01:00 (2023-10-07).

EDIT 2023-10-07 01:32 UTC: The maintenance is over. Deployments are now working again.

Part of our network infrastructure in Paris is unreachable. We are looking into it.

09:25 PM UTC: Network is back online. We are bringing back services which are not healthy.

09:30 PM UTC: Network is still flappy, we are on it.

10:06 PM UTC: Network seems stable. We are bringing back services which are not healthy.

10:37 PM UTC: We are bringing back services which are not healthy.

10:52 PM UTC: all services are back online. Good night.

The SSH Gateway will be unavailable on October 3rd, 2023 starting at 20:00 UTC. During that time, SSH access to services using either ssh -t ssh@sshgateway-clevercloud-customers.services.clever-cloud.com or using our CLI command clever ssh will be unavailable. Existing SSH connections through the gateway to services will be interrupted.

Once the maintenance is over, it is possible that some applications will need to be restarted to be able to be accessed through the SSH Gateway again.

The maintenance is planned to last less than 30 minutes.

EDIT 20:05 UTC: The maintenance is starting.

EDIT 20:27 UTC: The maintenance is now over. We are monitoring the results. You should now be able to access your services using the SSH gateway.

EDIT 20:27 UTC: Everything is working as intended. If you have any issues using the SSH gateway, you can try to redeploy your service and contact our support team.

Some pushes are not starting a new deployments. We are investigating.

EDIT 09:06 UTC: We implemented a fix and are monitoring the results. If you pushed new commits that didn't get deployed, you can either contact us through the support with your application id and the associated commit, or use our CLI with clever restart --commit <commit>.

EDIT 12:25 UTC: The incident is now resolved.

A maintenance on deployments and git repositories will happen at 22:00 UTC on October 2nd, 2023. During that time, deployments may be queued up for longer than usual. Git repositories will also be unavailable for a few minutes. Git pushes may fail to start new deployments or be rejected.

The maintenance is expected to last less than 30 minutes.

EDIT 22:00 UTC: The maintenance is starting

EDIT 22:35 UTC: The maintenance is mostly over. Deployments and git repositories are back since 15 minutes. We continue to make sure everything is running smoothly.

EDIT 23:05 UTC: Everything is back to normal since 22:20 UTC. The maintenance is over. Thanks for your patience.