Incident History

We are investigation on connectivity issues on the File System Buckets

EDIT 10:27 UTC: Connections should now be working again. It seemed that already established connections were also impacted and were slower than expected. This should now also be fixed.

EDIT 10:27 UTC: FS Buckets service is now fully operational .

We are currently experiencing issues on our deployment systems.

EDIT 13:25 UTC: Recovery takes longer than expected, we are still working on it.

EDIT 13:59 UTC: We are still working on fixing these issues.

EDIT 14:08 UTC: We are still having issues but deployments can start.

EDIT 14:41 UTC: Deployments performance has been back to normal for more than 15 minutes now. We are still watching the situation closely. If you have an issue, please contact us.

One of our hypervisor had a network issue for approximately 5 minutes.

Some of our internal services were impacted by this network issue and thus, automatic re-deployment of applications has been delayed.

Everything is back to normal, applications are currently finishing their redeployment.

Due to an ongoing maintenance from our provider, the logs system and a redis cluster of shared (and free) redis are unreachable. Logs may be lost. It should not last than 15 minutes according to them. A few minutes might be needed to restart the logs cluster.

Redis should be back as soon as the maintenance ends

EDIT 13:35 UTC: The maintenance is still ongoing

EDIT 13:50 UTC: The maintenance is over. Redis cluster is UP. Logs cluster is getting back UP. Logs should be saved but might not be directly available through the console

EDIT 14:30 UTC: The logs cluster is now fully operational too

One of our hypervisor has hard drive I/O failures. We are looking into it

EDIT 11:08 UTC: The server was shutdown a few minutes ago. Applications on it are being redeployed. Add-ons are currently unavailable

EDIT 11:52 UTC: We are still waiting for news from our provider regarding the hard drives issue

EDIT 21:20 UTC: Our provider is still working at finding the root cause of the issue

EDIT 2018-06-29 07:05 UTC: We received an answer from our provider and the server can't be brought back online. Databases will need migration. We are waiting an answer to know if we can access the disk in a read only mode to transfer the databases. If not, backups from the the 28th June will be used.

EDIT 2018-06-29 07:18 UTC: The disks can't be read. Backups will need to be used

Metrics are experiencing issues.

The logs collector needs to be restarted. Some logs might be lost for one to two minutes.

EDIT 22:00 UTC: Restart took approximately 30 seconds, most applications sent again the logs they couldn't send during that time

HV is down/unreachable. There seems to be a hardware problem. We are investigating it.

Some databases are unreachable.

EDIT 2018-06-18T23:25:00 UTC: Seems to be a malfunctioning fan. The server is still down for investigation. We are waiting for more informations from our hypervisor provider. EDIT 2018-06-19T00:37:00 UTC: The malfunctioning fans have been replaced. The server is up again. All the databases are up and running.

Some databases are unreachable.

EDIT 2018-06-18 16:29 UTC: The hypervisor is up again, the databases are getting back up.

Applications that were on this HV were redeployed on another one.

We have detected some network instabilities on one of our reverse proxy of the *.cleverapps.io domain, affecting the Paris zone. Our network provider has been notified.

EDIT 15:08 UTC: We are still waiting for our network provider to find the root cause of it.

EDIT 15-06-18 13:00 UTC: Instabilities have ceased since this morning. Everything should be back to normal

One of the nodes of the shared rabbitmq cluster went down. We are bringing it back

EDIT 10:40 UTC: The node has been restarted, we continue to monitor the situation.

EDIT 13:20 UTC: The cluster has been running fine since the incident

One of our add-on reverse proxy had to be restarted following an increasing rate of connections refuse. We will continue to monitor the situation closely

Our git repository will be shutdown for up to 15 minutes at 13:30 UTC, May 25th. Deployments will be shutdown and Git push / clone will be unavailable.

EDIT 13:30 UTC: The maintenance has begun. Deployments are shutdown (but are queued) and git repositories aren't available anymore.

EDIT 13:39 UTC: The maintenance is over, deployments and git repositories are available again

Some instances have troubles reaching VPN targets through our VPN service, we are investigating. Timeouts or unreachable routes are expected.

EDIT 09:45 UTC: We might have found why connections are hanging, we are currently doing some tests

EDIT 10:10 UTC: The tests worked fine and a fix has been deployed. All connections should have been restarted. If you still experience troubles with connecting to a particular service, please let us know at support@clever-cloud.com with the service you're trying to access

An operation maintenance is in progress on the storage backend of Metrics. Metrics are currently unavailable.

EDIT 14:40 UTC: Metrics are back since 14:15. Performance is gradually coming back to its usual level.

Deployments are having trouble to start or complete. We are working on it

EDIT 08:05 UTC: Deployments should be back to normal, we are keeping an eye on the situation.

EDIT 08:33 UTC: Some deployments still won't start

EDIT 09:00 UTC: Deployments should be back to normal again. We are still keeping an eye on the situation and cleaning up the remaining issues

EDIT 12:28 UTC: Again, some deployments are failing to finish even though they appear as successfully done in the logs. We are looking at it

EDIT 13:27 UTC: Deployments are going to be stopped to fully clean the system. It should not last more than 15 minutes. The maintenance is starting now.

EDIT 14:08 UTC: Deployments are available since 13:45 UTC. The maintenance period is over. We keep looking for everything to go back to normal

EDIT 16:30 UTC: Everything seems to be back to normal

We (or a client of us) were targeted by a DDoS attack starting at 10:05 UTC. We removed this IP from our front pool. The issue has been mitigated. We are still watching it.

At 8:13am Paris Time today, our hypervisor hv-par2-036 has been detected as unreachable.
A hard reboot has been requested to our hosting service.
Around 20 add-ons are impacted.

9:17am Paris Time: incident is fixed. All add-ons have recovered.

Network instabilities are affecting one of our reverse proxy, leading to packet / requests loss.

EDIT 13:50 UTC: Instabilities have stopped for 10 minutes now, we are still closely monitoring the situation.

The SSH Gateway asks for a password for PHP application instead of letting you connect. We are investigating the issue.

EDIT 08:00 UTC: A new version of the PHP image has been released. Redeploying your application should be enough to SSH again to the machine